Debezium Performance Tuning: Throughput, Latency, and Backpressure

A default Debezium installation handles most OLTP workloads without tuning. But when you're streaming from a high-write table — millions of rows per hour, burst write patterns, or large initial snapshots — the default configuration will bottleneck. This guide covers the tuning knobs that matter, where the bottlenecks actually occur, and how RisingWave handles the same problems without most of this complexity.

Understanding Debezium's Data Path

Before tuning, understand what happens to a change event between the database and Kafka:

1. The database writes a row change to the WAL (PostgreSQL) or binlog (MySQL).
2. Debezium's poller reads from the WAL stream in batches.
3. Events are transformed, schema-enriched, and serialized.
4. The Kafka producer buffers events in memory.
5. The producer flushes to Kafka brokers (respecting acks setting).
6. Kafka Connect commits the Debezium offset.

Each step can become the bottleneck depending on workload characteristics. The WAL read rate, serialization CPU, and Kafka producer throughput are the three most common limiting factors.

Key Debezium Throughput Parameters

Polling and Batching

{
  "max.batch.size": "8192",
  "max.queue.size": "16384",
  "poll.interval.ms": "500"
}

max.batch.size controls how many change events Debezium accumulates before passing them to the Kafka producer. The default is 2048. For high-throughput tables, increase this to 8192 or 16384 — larger batches amortize per-batch overhead.

max.queue.size is the in-memory queue between the WAL reader and the Kafka producer. It must be at least 2x max.batch.size. If the producer cannot keep up with the WAL reader, this queue fills and the WAL reader applies backpressure by pausing reads.

poll.interval.ms is the interval between WAL polls when there are no new events. The default of 500ms adds up to 500ms of latency for infrequent tables. For latency-sensitive pipelines on active tables, set it to 100ms or lower. For cold tables where latency matters less than resource consumption, 1000ms is fine.

Snapshot Fetch Size

{
  "snapshot.fetch.size": "10240"
}

During the initial snapshot, Debezium uses a JDBC cursor to read existing rows. snapshot.fetch.size controls the JDBC fetch batch size (default: varies by connector, typically 1024-2048). Increasing it to 10240 reduces round trips to the database during large table snapshots.

For a 500 million row table at 10240 rows per fetch: approximately 48,800 fetches. At 50ms per fetch, the full snapshot takes about 40 minutes. At the default 2048, it takes about 200 minutes.

Snapshot Parallelism

Debezium does not natively parallelize snapshots across partitions within a single connector. For very large tables, consider:

{
  "snapshot.select.statement.overrides": "public.orders",
  "snapshot.select.statement.overrides.public.orders": "SELECT * FROM public.orders WHERE created_at > '2024-01-01'"
}

This partitions the snapshot by a filter, letting you run multiple connectors with non-overlapping filters in parallel. Each connector captures its slice, and all feed into the same Kafka topic. This is not officially "parallel snapshot" but achieves the same effect.

Kafka Producer Tuning

The Debezium Kafka Connect worker uses the standard Kafka producer. These settings have the most impact on throughput:

# In the Kafka Connect worker config, or as producer.override.* in connector config
producer.override.acks=1
producer.override.linger.ms=20
producer.override.batch.size=131072
producer.override.compression.type=lz4
producer.override.buffer.memory=67108864

acks=1 acknowledges after the leader broker writes, without waiting for replicas. This halves latency for writes compared to acks=all but risks losing events if the leader crashes before replication. For CDC pipelines where the source database is the system of record, acks=1 is often acceptable since you can replay from the WAL.

linger.ms=20 holds the producer batch open for 20ms, allowing more events to accumulate into a single batch. This trades 20ms of additional latency for significantly higher throughput on bursty workloads.

compression.type=lz4 compresses Kafka messages. LZ4 is fast with reasonable compression ratios. CDC events (JSON or Avro) compress well — typically 3-5x reduction. This reduces both Kafka broker disk usage and network bandwidth.

batch.size=131072 (128KB) increases the producer batch size from the 16KB default. Combined with linger.ms, this allows batches to grow larger before being sent.

Common Bottlenecks and Diagnostics

Initial Snapshot of Large Tables

Symptoms: Connector in SNAPSHOT state for hours. Kafka consumer lag not growing (no streaming yet). Database CPU elevated from full table scan.

Diagnosis:

curl -s http://localhost:8083/connectors/orders-connector/status | jq '.tasks[0].state'
# Expect "RUNNING" during streaming, but for large snapshots, check logs:
docker logs debezium | grep "Snapshot step"

Mitigation: Increase snapshot.fetch.size. Schedule snapshots during off-peak hours. Consider using snapshot.mode=initial_only for the first connector, then a second connector with snapshot.mode=never for ongoing streaming.

Peak Write Bursts

Symptoms: Kafka consumer lag grows during write spikes, recovers during quiet periods. Connector RUNNING but perpetually lagged.

This is typically a Kafka producer throughput issue. Check:

# Kafka producer metrics via JMX
kafka.producer:type=producer-metrics,client-id=connector-producer-orders
  record-send-rate
  record-queue-time-avg
  request-latency-avg

If request-latency-avg spikes during write bursts, the broker is the bottleneck. If record-queue-time-avg spikes, the producer buffer is full — increase buffer.memory or reduce linger.ms.

Serialization CPU

For high-throughput connectors on small VMs, Avro serialization can saturate a single CPU core. Profile with:

jstack <connect-pid> | grep -A 5 "serializ"

Mitigation: Use JSON converter with schema disabled (schemas.enable=false) for lower serialization overhead. Or scale the Kafka Connect worker vertically.

Realistic Throughput Ranges

Based on typical deployments:

These ranges depend heavily on event size, network latency to Kafka, and broker hardware. A 1KB average event size will yield lower throughput than 200-byte events at the same configuration.

How RisingWave Handles Backpressure

RisingWave's CDC ingestion uses internal backpressure propagation — no Kafka producer to tune. When a downstream operator (aggregation, join) cannot keep up with ingestion rate, backpressure flows upstream through the streaming dataflow graph and automatically throttles the CDC reader.

-- Check current ingestion throughput in RisingWave
SELECT source_name, throughput_bytes, throughput_rows
FROM rw_source_stats;

The tradeoff: RisingWave's backpressure means the CDC reader slows down when the database is under load. The upstream database's replication slot accumulates more unread WAL. For databases with aggressive WAL retention policies, monitor slot lag:

-- On the PostgreSQL source
SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS slot_lag
FROM pg_replication_slots
WHERE slot_name = 'risingwave_slot';

Tuning Checklist

For Debezium:

• [ ] max.batch.size ≥ 8192 for high-write tables
• [ ] snapshot.fetch.size ≥ 10240 for large initial snapshots
• [ ] producer.override.linger.ms=20 and batch.size=131072 for throughput
• [ ] compression.type=lz4 enabled
• [ ] max.queue.size = 2x max.batch.size
• [ ] Monitor record-queue-time-avg and request-latency-avg via JMX

For RisingWave:

• [ ] Monitor rw_source_stats for ingestion throughput
• [ ] Monitor PostgreSQL replication slot lag during backpressure events
• [ ] Set checkpoint_frequency based on your recovery time tolerance

FAQ

What is the maximum throughput Debezium can achieve in practice? In benchmarks with tuned producers and co-located Kafka, Debezium has demonstrated 200,000+ events/sec for small events. For real-world production with network hops and mixed event sizes, 50,000-100,000 events/sec per connector is a more realistic ceiling before you need to shard across multiple connectors.

Does increasing max.batch.size increase latency? Slightly. Debezium accumulates up to max.batch.size events before flushing. In practice, high-write tables hit the batch size before any timeout, so observed latency does not increase. For low-write tables, the poll.interval.ms is the latency floor regardless of batch size.

Should I use Avro or JSON for high-throughput CDC? Avro is more compact and faster to deserialize at the consumer. For throughput-constrained pipelines, Avro with a schema registry is preferable. JSON with schemas.enable=false is simpler operationally but roughly 3-5x larger per message. At 50,000 events/sec, this difference in payload size materially affects Kafka disk and network costs.

What is Debezium's memory footprint during large snapshots? The in-memory queue (max.queue.size × average event size) dominates. At 16384 queue depth × 1KB per event = 16MB. The JDBC fetch buffer adds another snapshot.fetch.size × row width. For a table with wide rows (10KB+), a large snapshot.fetch.size can cause significant heap pressure. Monitor GC behavior during snapshots.

Can I throttle Debezium to reduce load on the source database? Not directly with a rate limiter. You can increase poll.interval.ms to reduce WAL read frequency, or use snapshot.delay.ms to delay the initial snapshot. For ongoing streaming, the WAL is read as fast as events are produced — Debezium does not idle-poll the database for streaming, so the load is minimal after the snapshot.