Postgres CDC to Iceberg: The Modern Approach

The modern approach to Postgres CDC to Iceberg is to eliminate the intermediate queue entirely. Instead of routing WAL events through Debezium, Kafka, and a Flink job, you connect a streaming database directly to Postgres logical replication, maintain state as a materialized view, and sink the results to Iceberg with upsert semantics. The pipeline shrinks from five moving parts to two.

This article walks through why that matters and how to build it.

What Postgres CDC to Iceberg Actually Requires

A production pipeline from Postgres to an Iceberg lakehouse has to satisfy more than "write rows to Parquet." The requirements that cause real problems in production are:

1. Upsert and delete semantics. Postgres is a transactional database. Every UPDATE and DELETE in the source must produce a correct, idempotent write to Iceberg. Append-only ingestion produces incorrect analytics.
2. Initial snapshot before streaming. The pipeline must perform a consistent full table read before entering CDC mode. Rows inserted during the snapshot window must not be lost or duplicated.
3. Schema evolution. Postgres schemas change. The pipeline must handle ALTER TABLE without a full re-snapshot or manual intervention.
4. End-to-end latency under 10 seconds. Typical analytical latency targets for operational Iceberg tables are in the single-digit second range. Anything slower means the lakehouse is not usable for near-real-time dashboards or ML feature freshness.
5. Operational simplicity. The number of components that can independently fail, need patching, or require tuning is a direct multiplier on operational cost.

The traditional stack and the RisingWave approach satisfy all five requirements, but with very different operational profiles.

Postgres Logical Replication: The Foundation

Both approaches read from the same source: the Postgres Write-Ahead Log (WAL).

Enabling CDC on a Postgres instance requires three steps:

-- In postgresql.conf
wal_level = logical

-- As superuser
ALTER SYSTEM SET wal_level = logical;
SELECT pg_reload_conf();

-- Create a replication slot
SELECT pg_create_logical_replication_slot('rw_slot', 'pgoutput');

-- Grant replication privilege to the CDC user
ALTER ROLE rw_user REPLICATION;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO rw_user;

With wal_level = logical, Postgres emits before and after images for every row change. The pgoutput plugin decodes these into a structured format that downstream consumers can parse. A replication slot acts as a durable cursor: it keeps WAL segments alive until the consumer confirms it has processed them, preventing data loss during consumer downtime.

WAL records contain the full operation type (INSERT, UPDATE, DELETE), the table identifier, the old row image (for UPDATE and DELETE), and the new row image (for INSERT and UPDATE). This is everything a CDC consumer needs to reconstruct current table state.

The Traditional Approach: Debezium + Kafka + Flink

The most common production pipeline for Postgres-to-Iceberg today uses three intermediaries.

Debezium runs as a Kafka Connect source connector. It connects to the Postgres replication slot, reads WAL events, and publishes CDC records to a Kafka topic per table. Each message is a JSON or Avro envelope containing the op field (c, u, d, r for create, update, delete, read), the before payload, and the after payload.

Kafka buffers the CDC stream. It decouples Debezium's read throughput from Flink's write throughput, provides replay semantics, and supports multiple consumers. In practice, managing Kafka topics for a Postgres workload with dozens of tables means managing dozens of topics, consumer group offsets, and retention policies.

Flink consumes the Kafka topics and writes to Iceberg. The Flink job must decode the Debezium envelope, route each event by op, apply upsert logic (tracking primary keys for equality deletes), and commit to Iceberg with exactly-once semantics. The Iceberg sink library handles the actual file writes, but the operator is responsible for Flink checkpoint storage, state backend sizing, and parallelism configuration.

A minimal production deployment for this stack includes:

That is five components before counting the object store, the orchestrator for compaction jobs, and the monitoring stack.

Latency in this stack accumulates at each hop: Debezium poll interval (typically 500ms-1s), Kafka producer batch delay, Kafka consumer poll interval, Flink checkpoint interval (often 30-60s for Iceberg commits), and Iceberg commit latency. End-to-end latency from a Postgres commit to a queryable Iceberg snapshot is typically 30 seconds to 5 minutes, depending on configuration.

Schema evolution is the most fragile part. When a column is added to a Postgres table, Debezium requires topic schema registry updates, the Flink job may need redeployment, and the Iceberg table DDL must be applied manually before or in sync with the new events arriving. Getting this wrong corrupts the table.

The RisingWave Approach: Direct CDC to Iceberg

RisingWave is a PostgreSQL-compatible streaming database. It connects directly to Postgres logical replication using a built-in CDC connector, maintains the current state of source tables as materialized views, and sinks those views to Iceberg with upsert semantics.

The entire pipeline is expressed in SQL, with no separate connector process, no message queue, and no stream processing job.

Step 1: Create the CDC Source

CREATE SOURCE pg_source WITH (
  connector = 'postgres-cdc',
  hostname = 'postgres-host',
  port = '5432',
  username = 'rw_user',
  password = '...',
  database.name = 'mydb',
  slot.name = 'rw_slot'
);

This statement opens a connection to the Postgres replication slot. RisingWave reads the replication stream using the pgoutput protocol directly, with no Debezium process in between.

Step 2: Create a Materialized View

CREATE MATERIALIZED VIEW orders_mv AS
  SELECT * FROM pg_source.public.orders;

RisingWave first performs a consistent snapshot of public.orders, then enters streaming CDC mode. The materialized view always reflects current table state. Every INSERT, UPDATE, and DELETE in Postgres propagates to orders_mv incrementally. No explicit upsert logic is required; the streaming database handles it.

Step 3: Sink to Iceberg

CREATE SINK orders_iceberg FROM orders_mv
WITH (
  connector = 'iceberg',
  type = 'upsert',
  primary_key = 'id',
  catalog.type = 'rest',
  catalog.uri = 'http://catalog:8181',
  database.name = 'analytics',
  table.name = 'orders'
);

The Iceberg sink writes the materialized view's current state to Iceberg with upsert semantics. Rows with matching primary keys are updated in place; deletes produce Iceberg delete files. The sink commits on a configurable checkpoint interval, keeping file counts bounded.

The Complete Pipeline

-- 1. Connect to Postgres WAL
CREATE SOURCE pg_source WITH (
  connector = 'postgres-cdc',
  hostname = 'postgres-host',
  port = '5432',
  username = 'rw_user',
  password = '...',
  database.name = 'mydb',
  slot.name = 'rw_slot'
);

-- 2. Materialize current table state
CREATE MATERIALIZED VIEW orders_mv AS
  SELECT * FROM pg_source.public.orders;

-- 3. Sink to Iceberg with upsert semantics
CREATE SINK orders_iceberg FROM orders_mv
WITH (
  connector = 'iceberg',
  type = 'upsert',
  primary_key = 'id',
  catalog.type = 'rest',
  catalog.uri = 'http://catalog:8181',
  database.name = 'analytics',
  table.name = 'orders'
);

Three SQL statements. No Debezium configuration, no Kafka topic provisioning, no Flink job JAR.

Initial Snapshot Handling

When RisingWave creates a CDC source, it begins with a consistent snapshot before entering streaming mode. The snapshot reads each source table at a fixed transaction snapshot identifier, ensuring no rows are missed and no duplicates are introduced when the streaming phase begins.

The transition from snapshot to streaming is handled internally. From the materialized view's perspective, the initial state is complete and correct before the first streaming event is applied. Data engineers do not need to manage a separate backfill job or reconcile snapshot data against incremental events.

In the Debezium approach, initial snapshot behavior is controlled by the snapshot.mode connector property. Managing the snapshot-to-streaming transition, especially for large tables, is a documented source of data loss when misconfigured.

Schema Evolution

When a column is added to a Postgres table, RisingWave detects the schema change from the WAL stream and propagates it automatically. The materialized view schema updates to include the new column, and the Iceberg sink adds the column to the Iceberg table schema on the next commit.

No manual DDL is required. No connector restart is needed.

In the Debezium + Flink stack, schema evolution requires coordinating multiple components:

• The Avro/JSON schema in the Kafka Schema Registry must be updated.
• The Flink job must be updated or restarted to handle the new field.
• The Iceberg table DDL must be executed against the catalog before or exactly in sync with the new events.

Any sequencing error produces malformed data. The typical mitigation is a full pipeline pause, DDL apply, and restart, which means downtime for the analytics table.

Latency Comparison

RisingWave's end-to-end latency from a Postgres COMMIT to a queryable Iceberg snapshot is typically 1 to 10 seconds, depending on the commit_checkpoint_interval setting. The default interval of 60 checkpoints at sub-second checkpoint frequency translates to roughly 1 minute for Iceberg commit latency, but the materialized view reflects changes within seconds and is itself queryable over the Postgres wire protocol.

The Debezium + Kafka + Flink stack typically achieves 30 seconds to 5 minutes end-to-end latency for Iceberg commits, with tighter latency requiring shorter Flink checkpoint intervals and more aggressive Iceberg sink configuration, which in turn increases small file pressure.

Architecture Comparison

Why the Component Count Matters

Each additional component in a data pipeline multiplies operational cost in a non-obvious way. A Debezium connector that loses its Kafka Connect worker needs a different runbook than a Kafka broker that loses a partition leader, which needs a different runbook than a Flink job that fails to checkpoint against its state backend. These failure modes do not compose.

RisingWave is open source under Apache 2.0 and distributed as a single binary or Kubernetes deployment. Failures are isolated to one system, observable through one monitoring interface, and recoverable through one operational model.

For teams whose core competency is data engineering rather than infrastructure operations, reducing the system boundary to one component is not a simplification for its own sake. It is a direct reduction in the number of things that can fail on a Saturday.

FAQ

Does RisingWave support multiple tables in a single CDC source?

Yes. A single CREATE SOURCE statement captures all tables in the specified Postgres database. Each table is accessible as pg_source.schema_name.table_name. You can create separate materialized views and Iceberg sinks per table, or join tables within a single materialized view before sinking.

What happens to the Iceberg table if RisingWave restarts mid-snapshot?

RisingWave persists its snapshot progress to durable storage. On restart, it resumes the snapshot from the last committed position rather than restarting from scratch. The Iceberg sink does not receive partial writes; data appears in Iceberg only after a full checkpoint commit.

How does RisingWave handle Postgres primary key updates?

An UPDATE that changes a primary key value is equivalent to a DELETE of the old key followed by an INSERT of the new key. RisingWave propagates this correctly through the materialized view: the old key is deleted from Iceberg via an equality delete file, and the new key is inserted as a new row.

Can I apply transformations before sinking to Iceberg?

Yes. The materialized view layer is where transformations live. You can filter rows, rename columns, cast types, aggregate, or join with other sources before the Iceberg sink. The sink always reads from the materialized view, not directly from the CDC source.

Is RisingWave production-ready for this use case?

RisingWave is in production at companies running large-scale Postgres deployments for operational analytics and lakehouse ingestion. The Postgres CDC connector and Iceberg sink are tested against major Iceberg catalog implementations including REST, AWS Glue, and Apache Hive. The project is open source under Apache 2.0, with commercial support available from RisingWave Labs.

RisingWave is a PostgreSQL-compatible streaming database designed for real-time analytics. It is open source under Apache 2.0. Get started in 5 minutes or join the RisingWave Slack community.