LSM Trees in Streaming Databases: How State Storage Works

LSM (Log-Structured Merge) trees are the storage engine behind streaming database state. RisingWave's Hummock engine and Flink's RocksDB backend both use LSM trees. They provide fast writes (append to memtable), efficient reads (with bloom filters), and background compaction.

Overview

Key Points

Streaming databases like RisingWave provide sub-second data freshness with SQL
PostgreSQL compatibility means existing tools and skills transfer directly
S3-based state storage provides durability, elastic scaling, and cost efficiency
Native CDC eliminates middleware for database-to-streaming pipelines
Iceberg integration bridges real-time serving with historical analytics

Architecture

The core pattern in all streaming architectures is:

Sources (CDC, Kafka) → Streaming Database (RisingWave) → Serving (PG protocol) + Storage (Iceberg)

This provides both real-time views (sub-second, via materialized views) and historical analytics (via Iceberg + Trino/DuckDB).

Implementation

-- The universal streaming pattern
CREATE SOURCE data_stream (...) WITH (connector='kafka', ...);
CREATE MATERIALIZED VIEW real_time_view AS SELECT ... FROM data_stream ...;
-- Query: SELECT * FROM real_time_view; (always fresh, sub-100ms)

Frequently Asked Questions

How does this relate to RisingWave?

RisingWave is a PostgreSQL-compatible streaming database that handles the processing and serving layers. It's open source (Apache 2.0), stores state on S3, and supports native CDC — making it the simplest path to production streaming.

Do I need Kafka for this?

Not always. RisingWave supports native CDC from PostgreSQL and MySQL without Kafka. For event-driven architectures with multiple consumers, Kafka remains valuable as a central event bus.