Debezium Initial Snapshot: How It Works and How to Speed It Up

The initial snapshot is the phase where Debezium reads your entire table before it can stream live changes. For small tables this takes seconds. For tables with 100M+ rows, it can take hours — and during that window, your replication slot is accumulating WAL you cannot consume yet.

What Happens During an Initial Snapshot

When Debezium starts for the first time (or after a reset), it needs a consistent point-in-time view of the data before it can switch to streaming mode.

For PostgreSQL, the snapshot process works like this:

1. Debezium opens a transaction with REPEATABLE READ isolation and records the current LSN (Log Sequence Number).
2. It runs a SELECT * FROM table query, paging through results.
3. After finishing all tables, it transitions to streaming from the LSN it recorded in step 1.

The critical insight: the replication slot is created before the snapshot begins, not after. This means WAL files accumulate during the entire snapshot duration. On a busy write-heavy table, a 6-hour snapshot can accumulate tens of gigabytes of WAL.

Snapshot Modes

Debezium exposes a snapshot.mode configuration that controls this behavior:

For most production deployments, schema_only is underused. If you have a downstream system that can be bootstrapped from a separate dump (e.g., pg_dump + restore), you can set snapshot.mode=schema_only and avoid the performance hit entirely.

# debezium-postgres-connector.properties
connector.class=io.debezium.connector.postgresql.PostgresConnector
snapshot.mode=schema_only
plugin.name=pgoutput
database.hostname=db.internal
database.port=5432
database.user=debezium
database.password=secret
database.dbname=prod
database.server.name=prod-server

Chunk-Based Snapshotting (Debezium 2.x)

Debezium 2.0 introduced incremental snapshots, which is the most significant performance improvement for large tables. Instead of one giant SELECT * transaction, Debezium splits the table into chunks and processes them one at a time.

The mechanism is based on the watermarking algorithm from the DBLog paper. It works as follows:

1. Debezium sends a signal (via a signals table) to start an incremental snapshot.
2. It reads a chunk of rows (e.g., 1000 rows by primary key range).
3. It interleaves chunk reads with processing live WAL events.
4. Changes that occur during the chunk read are de-duplicated using watermarks.

This approach means the snapshot can run alongside live streaming without blocking. It also means you can pause and resume.

-- Create the signals table (required for incremental snapshot)
CREATE TABLE debezium_signals (
  id VARCHAR(42) PRIMARY KEY,
  type VARCHAR(32) NOT NULL,
  data VARCHAR(2048) NULL
);

-- Trigger an incremental snapshot
INSERT INTO debezium_signals
VALUES ('ad-hoc-1', 'execute-snapshot', '{"data-collections": ["public.orders"]}');

Configure the connector to watch for signals:

signal.data.collection=public.debezium_signals
incremental.snapshot.chunk.size=8192

Tuning incremental.snapshot.chunk.size is the primary lever. Larger chunks mean fewer round trips but more memory pressure and longer individual queries. Start at 8192 and increase if your primary key selectivity allows it.

Why Snapshots Cause Replication Slot Bloat

This is the most operationally dangerous part of snapshots. During a long snapshot, your replication slot sits idle — it cannot advance its confirmed LSN until Debezium finishes reading and transitions to streaming.

PostgreSQL holds all WAL segments from that LSN forward. No vacuum can remove dead tuples whose transactions are newer than the slot's restart LSN. On a high-write database, this means:

• WAL disk usage grows linearly with snapshot duration.
• autovacuum becomes less effective.
• Table bloat increases.

Monitor this while a snapshot is running:

SELECT
  slot_name,
  confirmed_flush_lsn,
  restart_lsn,
  pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_behind,
  active
FROM pg_replication_slots;

If wal_behind exceeds your available disk space, you have a problem. Set a disk alert at 70% capacity before starting any large snapshot.

Practical Tuning Checklist

Tune snapshot.fetch.size: This controls the JDBC fetch size during the initial SELECT. Default is often too low.

snapshot.fetch.size=10240

Use parallel snapshots: Debezium 2.3+ supports parallel snapshot workers. Each worker handles a subset of tables.

snapshot.max.threads=4

Filter tables aggressively: Only snapshot what you need. Use table.include.list to exclude large historical tables.

table.include.list=public.orders,public.customers

Consider exported mode for very large tables: Run pg_dump externally, restore to your target, then use exported mode to start streaming from the dump's LSN.

How RisingWave Handles the Same Problem

RisingWave uses the Debezium Embedded Engine internally for its PostgreSQL CDC source. This means it faces the same snapshot mechanics — but exposes them differently.

When you create a CDC source in RisingWave, it runs the initial snapshot automatically and then transitions to streaming. You don't configure snapshot modes directly; instead, you control behavior through table options.

-- RisingWave CDC source (handles snapshot + streaming automatically)
CREATE SOURCE pg_source WITH (
  connector = 'postgres-cdc',
  hostname = 'db.internal',
  port = '5432',
  username = 'rwuser',
  password = 'secret',
  database.name = 'prod',
  schema.name = 'public',
  table.name = 'orders'
);

-- Create the materialized view that queries from the source
CREATE MATERIALIZED VIEW orders_mv AS
SELECT * FROM pg_source;

For large tables, RisingWave's snapshot runs in the background while you can already query partially materialized results. The BACKFILL mechanism in RisingWave is analogous to Debezium's incremental snapshot — it processes chunks and merges live events.

The key operational difference: RisingWave stores checkpoints to S3, so a snapshot that gets interrupted can resume from the last checkpoint rather than starting over. This is a significant operational advantage for very large tables where a full restart would be costly.

FAQ

How long will my initial snapshot take? Rough estimate: 1M rows/minute per table on a well-indexed table over a local network connection. A 100M-row table takes roughly 90-120 minutes. Actual performance depends on row width, network latency, snapshot.fetch.size, and database load.

Can I run Debezium during business hours while snapshoting a large table? Yes, but carefully. The snapshot query holds a REPEATABLE READ transaction open. This does not block writes, but it does prevent PostgreSQL from reclaiming dead row versions (affecting autovacuum). Schedule large snapshots during low-write periods.

What happens if the snapshot fails halfway through? With initial mode, Debezium restarts the snapshot from scratch. With incremental snapshots and a signals table, progress is tracked and it can resume. For large tables, always prefer incremental snapshots.

Why does my disk fill up during snapshots? The replication slot holds WAL from the snapshot start LSN forward. On a high-write database, this accumulates fast. Monitor pg_replication_slots.restart_lsn and ensure you have at least 2x your expected WAL generation rate in free disk space before starting.

Should I use schema_only if I can bootstrap data another way? Yes. If you can load historical data via COPY, pg_dump, or a data platform tool, schema_only mode gives you streaming-only CDC with zero snapshot risk. This is often the right call for tables over 50M rows.