Replacing Debezium in Your Real-Time Analytics Stack

For teams running Debezium → Kafka → Flink or Spark solely for analytics — not fan-out — RisingWave can replace the entire pipeline with a single system. You keep the CDC reliability. You eliminate Kafka, Kafka Connect, and the separate stream processor. The trade-off: if you need fan-out to multiple consumers, do not replace Debezium.

Who This Migration Is For

This article targets a specific architecture: Debezium publishing to Kafka, and a single analytics consumer (Flink, Spark Streaming, or a custom consumer) reading from Kafka to power dashboards or serve real-time queries.

If you have multiple Kafka consumers reading those topics for different purposes — Elasticsearch indexing, a data lake loader, a notification service — this migration is not for you. Kafka's fan-out value is real in that case, and Debezium is the right tool.

If your only Kafka consumer is analytics, keep reading.

The Before Architecture

Here is what the typical pipeline looks like before migration:

PostgreSQL
    ↓
Debezium (Kafka Connect plugin)
    ↓
Kafka (managed or self-hosted)
    ↓
Flink / Spark Streaming / custom consumer
    ↓
Analytics layer (BI tools, dashboards)

Each component has its own operational surface:

• Kafka Connect workers with connector config, restart policies, and lag monitoring
• Kafka brokers with replication factor, retention, and partition management
• A stream processing cluster (Flink JobManager + TaskManagers, or Spark executors)
• Schema registry (often required for Debezium + Avro)
• Offset tracking and recovery procedures

This is the right architecture when you need it. The problem is that many teams build it because "that's how you do streaming," then spend 40% of their engineering time keeping it running.

The After Architecture

PostgreSQL
    ↓
RisingWave (CDC + SQL processing + serving)
    ↓
Analytics layer (BI tools, dashboards via PostgreSQL protocol)

RisingWave reads directly from the PostgreSQL replication slot. Internally, it uses the Debezium Embedded Engine for log capture — the same battle-tested library. Transformations that lived in Flink SQL or Spark become materialized views. Your BI tools connect to RisingWave over the standard PostgreSQL wire protocol.

What You Can Replace

Debezium Connector Config → RisingWave CREATE SOURCE

Before (Debezium connector JSON):

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db.internal",
    "database.port": "5432",
    "database.user": "replicator",
    "database.password": "secret",
    "database.dbname": "ecommerce",
    "database.server.name": "ecommerce",
    "table.include.list": "public.orders,public.order_items",
    "plugin.name": "pgoutput"
  }
}

After (RisingWave SQL):

CREATE SOURCE pg_source WITH (
    connector = 'postgres-cdc',
    hostname = 'db.internal',
    port = '5432',
    username = 'replicator',
    password = 'secret',
    database.name = 'ecommerce',
    slot.name = 'risingwave_slot'
);

CREATE TABLE orders (
    id BIGINT PRIMARY KEY,
    customer_id BIGINT,
    status VARCHAR,
    total NUMERIC,
    created_at TIMESTAMPTZ
) FROM pg_source TABLE 'public.orders';

CREATE TABLE order_items (
    id BIGINT PRIMARY KEY,
    order_id BIGINT,
    product_id BIGINT,
    quantity INT,
    unit_price NUMERIC
) FROM pg_source TABLE 'public.order_items';

Flink SQL Jobs → RisingWave Materialized Views

Before (Flink SQL):

-- Flink: revenue by customer over a sliding window
SELECT
    customer_id,
    SUM(total) AS total_revenue,
    COUNT(*) AS order_count
FROM orders
WHERE status = 'completed'
GROUP BY customer_id;

After (RisingWave materialized view — same SQL, runs continuously):

CREATE MATERIALIZED VIEW customer_revenue AS
SELECT
    customer_id,
    SUM(total) AS total_revenue,
    COUNT(*) AS order_count
FROM orders
WHERE status = 'completed'
GROUP BY customer_id;

The key difference: in Flink, you deploy a job. In RisingWave, you run a SQL statement. The view stays current automatically.

Multi-table joins across CDC streams

Before (Flink): joining two Kafka topics requires managing changelog semantics, watermarks, and state backends.

After (RisingWave):

CREATE MATERIALIZED VIEW order_summary AS
SELECT
    o.id AS order_id,
    o.customer_id,
    o.status,
    SUM(oi.quantity * oi.unit_price) AS line_total,
    COUNT(oi.id) AS item_count
FROM orders o
JOIN order_items oi ON o.id = oi.order_id
GROUP BY o.id, o.customer_id, o.status;

RisingWave handles the join state internally. No Kafka topics to manage, no state backend configuration.

When to Keep Debezium in Your Stack

Be clear about your pipeline's scope before migrating.

Fan-out to multiple consumers. If Kafka topics feed Elasticsearch, a data lake, a notification service, and analytics simultaneously, Debezium + Kafka is the right architecture for distribution. The common pattern is to keep Debezium + Kafka for fan-out, and add RisingWave as the analytics consumer on those same Kafka topics.

Non-SQL sinks. If a downstream system needs raw CDC events delivered directly (e.g., for event sourcing or audit logging), Kafka Connect sinks handle this well. RisingWave can coexist: Debezium publishes to Kafka, RisingWave subscribes for analytics, and other Kafka consumers receive raw events.

Existing Kafka infrastructure. If your team already operates Kafka and the marginal cost of running a Debezium connector is low, there is no urgency to replace it. RisingWave integrates as an additional layer, consuming from Kafka topics, rather than requiring a full stack replacement.

Migration Steps

Step 1: Audit your Kafka consumers. List every consumer reading CDC topics. If there is more than one, evaluate whether each is doing analytics or something else.

Step 2: Identify the analytics consumer. Determine exactly what transformations the Flink or Spark job performs. List every aggregation, join, and filter. These become materialized views.

Step 3: Set up RisingWave in parallel. Run RisingWave alongside the existing stack. Create the CDC source and tables. Create materialized views for each transformation.

Step 4: Validate outputs. Compare query results between RisingWave and the existing analytics layer. Run both in parallel for at least 24 hours to validate consistency.

Step 5: Cut over BI tools. Update dashboards and BI tool connection strings to point to RisingWave (PostgreSQL protocol, port 4566 by default).

Step 6: Decommission Kafka and Debezium. Once traffic is validated on RisingWave, shut down Kafka Connect, the Debezium connector, the Flink or Spark cluster, and the Kafka brokers (if they were only serving this pipeline).

Monitoring After Migration

RisingWave exposes metrics via a built-in Prometheus endpoint. Key metrics to watch:

-- Check CDC source lag
SELECT * FROM rw_source_backfill_info;

-- Check materialized view freshness
SELECT * FROM rw_materialized_views;

-- Check source connector status
SELECT connector_name, status FROM rw_sources;

For production deployments, set alerts on CDC consumer lag and materialized view refresh errors.

FAQ

Q: Will RisingWave's initial snapshot cause downtime? No. RisingWave performs an initial table snapshot while continuing to buffer change events from the replication log. Once the snapshot completes, buffered events are applied. This process is online and does not require locking or source database downtime.

Q: How does RisingWave handle schema changes in the source table? RisingWave handles additive changes (new columns) automatically. Destructive changes (dropping or renaming columns) require manual intervention: drop the affected table in RisingWave, update the schema, and recreate the source. This is comparable to how Debezium handles schema evolution with schema registry updates.

Q: What if my Flink jobs use custom UDFs? RisingWave supports user-defined functions via Python, JavaScript, or Java. Standard UDFs in Flink SQL can usually be rewritten as SQL functions. Complex UDFs may require a UDF adapter.

Q: Do I need to keep Kafka for anything? Only if other systems depend on those Kafka topics for non-analytics purposes. If the analytics pipeline was the only Kafka consumer, you can decommission Kafka entirely after migration.

Q: How does latency compare? End-to-end latency in the Debezium → Kafka → Flink pipeline is typically 1–5 seconds depending on Kafka consumer poll intervals and Flink checkpoint intervals. RisingWave processes change events in milliseconds, typically achieving sub-second latency from database commit to materialized view update.