Data Mesh and Stream Processing: Real-Time Domain Data Products
Serverless stream processing uses ephemeral functions (AWS Lambda, Cloud Functions) triggered by events, instead of dedicated streaming clusters. It's simpler for low-volume workloads but limited for stateful processing.
Serverless vs Dedicated Streaming
| Aspect | Serverless (Lambda) | Dedicated (RisingWave/Flink) |
| State | External (DynamoDB) | Built-in |
| Latency | Cold starts (100ms-1s) | Always-on (sub-100ms) |
| Throughput | Concurrency limits | Horizontal scaling |
| Joins/Windows | Manual, complex | Built-in SQL |
| Cost at low volume | Very cheap | Fixed baseline |
| Cost at high volume | Expensive | More efficient |
When to Use Serverless
- Low-volume event processing (<100 events/sec)
- Simple transformations (filter, route, enrich)
- Stateless processing (no aggregations/joins)
- Pay-per-use cost model preferred
When to Use Dedicated Streaming
- Stateful processing (aggregations, joins, windows)
- High throughput (>1000 events/sec)
- Sub-100ms latency requirements
- Complex multi-source pipelines
Frequently Asked Questions
Can I use Lambda for stream processing?
Yes, for simple, stateless, low-volume workloads. For stateful processing (aggregations, joins), Lambda requires external state management (DynamoDB) which adds complexity and latency. Use a streaming database instead.
