Real-Time Lakehouse Architecture: Single System vs Multi-Component

A real-time lakehouse architecture combines open table formats like Apache Iceberg with sub-minute data freshness, serving both analytical queries and streaming consumers from a single storage layer. The central architectural choice is whether to achieve this with a coordinated multi-component pipeline or a unified streaming database.

What Is a Real-Time Lakehouse?

A data lakehouse merges two historically separate paradigms. The data warehouse provides strong consistency, structured schemas, and fast query performance. The data lake provides cheap scalable storage, schema flexibility, and support for diverse workloads. Open table formats — Apache Iceberg, Delta Lake, Apache Hudi — made the merge possible by adding ACID transactions and schema evolution on top of object storage.

The "real-time" qualifier adds a freshness requirement. A traditional lakehouse might run hourly or daily batch loads. A real-time lakehouse targets sub-minute latency: data committed to the source database should be queryable in the lakehouse within seconds to a few minutes.

This freshness constraint is what forces the architectural decision. Batch ETL pipelines cannot meet it. You need a streaming infrastructure. The question is how many systems that streaming infrastructure requires.

Architecture 1: The Multi-Component Pipeline

The dominant pattern in production today assembles several best-of-breed components into a data pipeline.

Components:

1. Change data capture (CDC) source — Debezium or a managed CDC service reads the MySQL/PostgreSQL write-ahead log and emits row-level change events
2. Apache Kafka — Message broker that buffers change events, decouples producers from consumers, and provides replay capability
3. Apache Flink or Spark Streaming — Stream processing engine that consumes Kafka topics, applies transformations and aggregations, and writes results downstream
4. Apache Iceberg — Open table format that receives committed data, maintains table metadata, and serves readers
5. Query engine — Trino, Amazon Athena, or Spark SQL connects to Iceberg for ad-hoc analytical queries
6. Apache Airflow — Orchestration layer that schedules compaction jobs, snapshot expiration, and orphan file cleanup on Iceberg tables
7. Monitoring stack — Prometheus, Grafana, or a commercial APM product covers each component independently

The full stack is 5 to 7 components depending on whether CDC tooling and monitoring are counted separately. Each component is independently scalable and independently deployable. Each also requires independent expertise.

How data flows:

Source database writes a row. Debezium captures the WAL entry and publishes a CDC event to a Kafka topic. A Flink job consumes that topic, applies any business logic, and writes to an Iceberg table via the Iceberg Flink connector. Airflow runs a nightly compaction job to merge small files. Trino queries the Iceberg table metadata catalog and reads Parquet files from S3.

This architecture is mature. It has extensive tooling, large communities, and documented operational playbooks.

Architecture 2: The Single-System Approach

An emerging alternative collapses the ingestion and transformation layers into a single streaming database.

Components:

1. Source database with CDC — Same as before: MySQL, PostgreSQL, or another OLTP system with change data capture enabled
2. RisingWave — A PostgreSQL-compatible streaming database that ingests CDC events or Kafka topics directly, applies SQL-defined transformations via materialized views, and writes results to Iceberg using native Iceberg sink connectors
3. Apache Iceberg — Same open table format, same S3-backed storage
4. Query engine — Trino or Athena still reads from Iceberg for ad-hoc analytics

The component count drops from 7 to 4, and the ingestion pipeline itself is 3 components: source, RisingWave, and Iceberg.

How data flows:

RisingWave connects directly to the source database using built-in CDC connectors (MySQL, PostgreSQL, MongoDB, and others). Materialized views define transformations in standard SQL. RisingWave maintains those views incrementally as new events arrive and commits results to Iceberg tables on a configurable interval. Compaction is handled internally. Trino or Athena reads from Iceberg for analytical queries.

RisingWave is open source under the Apache 2.0 license. It is written in Rust and uses a disaggregated storage architecture where compute is separated from state storage on S3, enabling independent scaling of compute and storage resources.

The Consistency Problem

Consistency is the hardest problem in multi-component pipelines and the one most often underestimated at design time.

In the multi-component architecture, data exists in three distinct states simultaneously: raw events in Kafka, intermediate state in Flink checkpoints, and committed rows in Iceberg. These three states are maintained by three different systems with three different failure and recovery models.

If a Flink job crashes between a checkpoint and an Iceberg commit, the recovery process must replay Kafka messages from the last checkpoint offset and re-attempt the Iceberg write. If Kafka retention has expired or the checkpoint is corrupted, data loss is possible. Achieving exactly-once semantics requires careful alignment of Kafka consumer group offsets, Flink checkpoint intervals, and Iceberg commit intervals.

In the single-system architecture, RisingWave owns the full path from CDC ingestion to Iceberg commit. Internal checkpointing and Iceberg commits are coordinated by the same engine. There is no gap between "what the streaming engine processed" and "what Iceberg received." Iceberg commits are atomic. Recovery restarts from the last consistent internal checkpoint and re-issues the Iceberg write.

This does not mean the single-system approach eliminates all failure modes. It means the failure domain is smaller and the recovery logic is managed by a single system rather than distributed across three.

Latency Comparison

End-to-end latency in the multi-component pipeline accumulates across stages.

Debezium-to-Kafka introduces a small but nonzero buffering delay. Flink processing adds checkpoint interval latency, typically 30 to 60 seconds in production configurations that prioritize throughput. Iceberg commits introduce additional latency because small frequent commits create many small files, degrading query performance; operators therefore commit in batches, often every 60 to 120 seconds.

Typical end-to-end latency in a production multi-component pipeline: 30 to 120 seconds.

In the RisingWave path, the ingestion-to-Iceberg latency is controlled by a single commit interval parameter. RisingWave's internal checkpoint mechanism is designed for low-latency incremental processing. Iceberg sink commit intervals of 10 to 30 seconds are typical in production deployments.

Typical end-to-end latency with RisingWave: 5 to 30 seconds.

The latency difference is not always operationally significant. For dashboards refreshed every 5 minutes, both architectures deliver acceptable freshness. For fraud detection, anomaly alerting, or real-time personalization, the difference between 10 seconds and 90 seconds can determine whether the architecture meets its SLA.

Schema Management

Schema evolution in the multi-component architecture requires synchronized changes across multiple systems.

When a source table adds a column, the change must propagate through: the CDC connector configuration, the Kafka topic schema (if using Schema Registry), the Flink job's type definitions or dynamic schema handling, and the Iceberg table schema. Each of these systems has its own schema management tooling and its own failure modes. A schema change that succeeds in three of four layers and fails in the fourth can corrupt downstream data or halt the pipeline.

RisingWave handles schema evolution through Iceberg's native schema evolution API. When a source schema changes, RisingWave detects the change through the CDC stream and can propagate it to Iceberg automatically. There is one schema management boundary to reason about rather than four.

Architecture Comparison

Text Architecture Diagrams

Multi-component pipeline:

[Source DB]
    |
    | (WAL / CDC)
    v
[Debezium] --> [Kafka Topics] --> [Flink Job] --> [Iceberg Tables] --> [Trino / Athena]
                                                        ^
                                              [Airflow Compaction]

Single-system pipeline:

[Source DB]
    |
    | (CDC / Kafka)
    v
[RisingWave] --> [Iceberg Tables] --> [Trino / Athena]
(materialized
 views in SQL)

The single-system diagram removes the Kafka broker, Flink cluster, and Airflow orchestrator from the ingestion path. Kafka may still exist upstream as an event bus from other systems; in that case RisingWave consumes it directly without an intermediate Flink layer.

When Multi-Component Still Makes Sense

The multi-component architecture is not obsolete. It remains the right choice under specific conditions.

Petabyte-scale throughput. When daily data volumes exceed hundreds of terabytes, Flink's horizontal scalability and fine-grained resource partitioning are difficult to match with any single-system solution.

Mixed team ownership. When the team responsible for Kafka is different from the team responsible for transformations, which is different from the team responsible for the lakehouse layer, the component boundaries map to organizational boundaries. Consolidating into a single system may create ownership ambiguity.

Existing investment. If a team has invested years in Flink expertise, custom connectors, and operational tooling, the switching cost may outweigh the simplification benefit.

Regulatory isolation. Some compliance frameworks require physical data boundaries between processing stages. A multi-component architecture makes those boundaries explicit.

For teams starting a new real-time lakehouse project at moderate scale (up to tens of terabytes per day), the single-system approach offers a faster path to production with a smaller operational burden.

Decision Framework

FAQ

Q: Can RisingWave replace Kafka entirely in a real-time lakehouse?

Not in most architectures. Kafka serves as a durable, replayable event bus for multiple downstream consumers, not just the lakehouse pipeline. RisingWave can consume directly from Kafka or from CDC sources. It does not replace Kafka's role as a shared event distribution layer. What it replaces is the Flink layer that would otherwise sit between Kafka and Iceberg.

Q: Does the single-system approach lock you into RisingWave?

The data storage layer remains Iceberg on S3, which is entirely open. If you migrate away from RisingWave, your Iceberg tables remain intact and queryable by Trino, Athena, Spark, or any other Iceberg-compatible engine. The lock-in risk is limited to transformation logic defined as materialized views, which would need to be rewritten for a different processing engine.

Q: How does exactly-once delivery work when RisingWave writes to Iceberg?

RisingWave implements exactly-once semantics by coordinating its internal checkpoint barrier protocol with Iceberg's atomic commit mechanism. When a checkpoint completes successfully, the corresponding Iceberg commit is issued. On recovery after failure, RisingWave restores to the last completed checkpoint and re-issues any Iceberg writes that did not commit. This is analogous to Flink's two-phase commit to Iceberg, but managed within a single system rather than across a Flink cluster and an external commit coordinator.

Q: What happens to ad-hoc query performance as Iceberg tables grow?

Iceberg table performance depends on file size distribution and partitioning strategy, not on the ingestion path. Whether data arrives via Flink or RisingWave, small files accumulate and must be compacted. RisingWave handles compaction internally for tables it manages. For very large tables with complex access patterns, explicit Iceberg compaction strategies using Spark or a dedicated compaction service may still be warranted.

Q: Is this architecture suitable for mixed workloads where some consumers need real-time and others need batch?

Yes. Iceberg supports concurrent readers at different snapshot versions. A real-time consumer (reading RisingWave materialized views directly) and a batch consumer (running Trino queries against Iceberg snapshots) can coexist on the same data without conflict. The single-system approach does not constrain the downstream consumer pattern.