Stream Processing Performance Tuning Guide
Streaming performance depends on parallelism, state management, checkpoint configuration, and resource allocation. This guide covers the key tuning parameters for RisingWave, with principles applicable to any streaming system.
Performance Tuning Checklist
| Area | What to Tune | Impact |
| Parallelism | Number of streaming fragments | Throughput |
| Checkpoint interval | Frequency of state snapshots | Latency vs durability |
| Memory | Compute node memory allocation | State cache hit rate |
| Compaction | Background compaction workers | Query latency |
| Source partitions | Kafka partitions / CDC slots | Ingestion throughput |
Key Principles
1. Match Parallelism to Source Partitions
Kafka topic with 12 partitions → 12+ parallel consumers in RisingWave
Under-parallelism wastes source capacity. Over-parallelism wastes compute.
2. Bound Your State
-- UNBOUNDED (state grows forever):
SELECT user_id, COUNT(*) FROM events GROUP BY user_id;
-- BOUNDED (state limited to 24h window):
SELECT user_id, COUNT(*) FROM events
WHERE ts > NOW() - INTERVAL '24 hours' GROUP BY user_id;
3. Use Cascading Views for Complex Pipelines
-- Instead of one massive view, cascade:
CREATE MATERIALIZED VIEW stage_1 AS SELECT ... FROM raw_events WHERE ...;
CREATE MATERIALIZED VIEW stage_2 AS SELECT ... FROM stage_1 JOIN dim_table ...;
CREATE MATERIALIZED VIEW final AS SELECT ... FROM stage_2 GROUP BY ...;
4. Monitor Key Metrics
- Barrier latency (RisingWave) / Checkpoint duration (Flink)
- Memory usage per compute node
- Source-to-output latency (end-to-end)
Frequently Asked Questions
What is the most common streaming performance issue?
Unbounded state — aggregations without time bounds cause state to grow indefinitely, eventually exhausting memory. Always add WHERE ts > NOW() - INTERVAL to limit state.
How do I measure streaming performance?
Key metrics: end-to-end latency (source event time to query result), throughput (events/second), checkpoint/barrier latency, and memory usage. RisingWave exposes these via Prometheus metrics.
