Stream Processing on Kubernetes: Architecture Guide

Kubernetes is the standard deployment platform for stream processing in 2026. RisingWave, Apache Flink, and Kafka all run well on K8s. This guide covers architecture patterns, resource management, and operational best practices for streaming on Kubernetes.

Streaming on Kubernetes Architecture

┌─────────── Kubernetes Cluster ───────────┐
│                                          │
│  ┌─── RisingWave ────┐  ┌─── Kafka ──┐  │
│  │ Frontend (SQL)     │  │ Brokers    │  │
│  │ Compute Nodes (3+) │  │ (StatefulSet)│ │
│  │ Compactor          │  └────────────┘  │
│  │ Meta (etcd)        │                  │
│  └────────────────────┘  ┌── Grafana ──┐ │
│                          │ Monitoring  │  │
│  ┌─── S3 (external) ─┐  └─────────────┘ │
│  │ State storage      │                  │
│  └────────────────────┘                  │
└──────────────────────────────────────────┘

RisingWave on Kubernetes

# Helm chart deployment
helm repo add risingwave https://risingwavelabs.github.io/helm-charts/
helm install risingwave risingwave/risingwave \
  --set metaStore.etcd.enabled=true \
  --set stateStore.s3.enabled=true \
  --set stateStore.s3.bucket=my-rw-state

Key Considerations

Concern	Best Practice
State storage	Use S3 (not local PV) for RisingWave state
Resource requests	Set CPU/memory requests = limits for predictable scheduling
Pod disruption	Set PDB to prevent simultaneous evictions
Scaling	Scale compute nodes independently from meta/compactor
Monitoring	Prometheus + Grafana for metrics

Frequently Asked Questions

Should I use local PersistentVolumes for streaming state?

No. Use S3 or compatible object storage. Local PVs tie state to specific nodes, making recovery and scaling difficult. S3-based state (RisingWave, Flink 2.0 ForSt) enables pod rescheduling without state migration.

How do I scale streaming on Kubernetes?

Scale compute nodes with HPA or manually. With S3-based state, new pods read state from S3 immediately — no state redistribution needed.