Migrating from Lambda to Kappa architecture means consolidating your batch and streaming pipelines into one streaming SQL layer. For e-commerce platforms, this eliminates duplicated business logic, closes consistency windows between dashboards and source data, and dramatically reduces API latency — SHOPLINE achieved a 76.7% reduction after making this transition with RisingWave.
The Problem: Lambda Architecture Debt in E-Commerce
Lambda architecture made sense when streaming engines were immature. The idea was pragmatic: run a batch pipeline for accuracy and a streaming pipeline for speed, then merge the results at query time. In practice, this creates a compounding maintenance problem.
In e-commerce, the pain surfaces in predictable places:
Duplicated GMV logic. Your batch job computes gross merchandise value from Redshift or BigQuery. Your streaming job computes the same metric in Flink or Spark Streaming. When the product team changes how discounts are applied to GMV, both pipelines need updating — and they rarely get updated at the same time.
Consistency windows. A merchant checks their dashboard at 9:47 AM. The batch job last ran at 9:30 AM. The streaming counter includes events the batch job hasn't processed yet. The numbers disagree. Support tickets follow.
Operational complexity. Two separate engineering teams, two monitoring stacks, two incident runbooks — for what is conceptually one data transformation.
Slow API responses. Dashboard APIs that fan out to both the warehouse (for historical data) and the streaming store (for recent data) carry the latency of both round trips.
How Streaming SQL Solves This
Kappa architecture, coined by LinkedIn engineer Jay Kreps, proposes a simpler model: treat everything as a stream, reprocess using the same streaming logic for both real-time and historical data, and maintain a single codebase.
RisingWave — a PostgreSQL-compatible streaming database — makes Kappa architecture practical for teams that know SQL. Instead of maintaining Flink jobs and dbt models separately, you write CREATE MATERIALIZED VIEW statements that RisingWave keeps incrementally up to date as new events arrive from Kafka.
The key properties that enable this:
- Incremental computation: RisingWave never recomputes a materialized view from scratch. Each new event triggers a minimal delta update.
- PostgreSQL wire protocol: Downstream applications connect to RisingWave the same way they connect to Postgres — no custom clients.
- Kafka replay for reprocessing: Because Kafka retains history, you can drop a materialized view, fix the logic, and recreate it — RisingWave reprocesses from the committed offset automatically.
Migration Guide: Lambda to Kappa in Four Steps
Step 1: Data Source Setup
Before you migrate, map every source your batch pipeline reads. For most e-commerce platforms, this includes order events, inventory updates, and customer activity — all flowing through Kafka. Connect RisingWave to the same Kafka topics your streaming pipeline already consumes:
CREATE SOURCE orders (
order_id VARCHAR,
merchant_id VARCHAR,
customer_id VARCHAR,
channel VARCHAR, -- 'web', 'mobile', 'pos', 'marketplace'
payment_method VARCHAR, -- 'card', 'alipay', 'paypal', 'cod'
subtotal NUMERIC,
discount NUMERIC,
total NUMERIC,
status VARCHAR,
created_at TIMESTAMPTZ
) WITH (
connector = 'kafka',
topic = 'ecommerce.orders',
properties.bootstrap.server = 'kafka:9092',
scan.startup.mode = 'earliest'
) FORMAT PLAIN ENCODE JSON;
CREATE SOURCE inventory_events (
event_id VARCHAR,
sku_id VARCHAR,
merchant_id VARCHAR,
delta INT,
event_type VARCHAR, -- 'sale', 'restock', 'adjustment'
occurred_at TIMESTAMPTZ
) WITH (
connector = 'kafka',
topic = 'ecommerce.inventory',
properties.bootstrap.server = 'kafka:9092',
scan.startup.mode = 'earliest'
) FORMAT PLAIN ENCODE JSON;
The scan.startup.mode = 'earliest' setting tells RisingWave to reprocess Kafka history from the beginning — this is how you backfill your Kappa layer without touching your batch pipeline.
Step 2: Core Materialized View
Translate your most critical batch transformation into a materialized view. Start with the metric that causes the most consistency pain — usually GMV or order counts:
CREATE MATERIALIZED VIEW mv_merchant_hourly_stats AS
SELECT
merchant_id,
channel,
WINDOW_START AS hour_start,
WINDOW_END AS hour_end,
COUNT(*) AS order_count,
SUM(total) AS gmv,
SUM(discount) AS discount_amount,
AVG(total) AS aov,
COUNT(DISTINCT customer_id) AS unique_buyers,
COUNT(CASE WHEN status = 'refunded'
THEN 1 END) AS refund_count
FROM TUMBLE(orders, created_at, INTERVAL '1 HOUR')
GROUP BY merchant_id, channel, WINDOW_START, WINDOW_END;
Run this in parallel with your existing batch pipeline. Compare outputs. When they agree (they will, once you've matched the business logic), you have your cutover candidate.
Step 3: Aggregations / Alerts
Add higher-level aggregations that your batch pipeline computed with daily jobs — these become additional materialized views:
CREATE MATERIALIZED VIEW mv_merchant_daily_summary AS
SELECT
merchant_id,
DATE_TRUNC('day', hour_start) AS day,
SUM(order_count) AS daily_orders,
SUM(gmv) AS daily_gmv,
SUM(discount_amount) AS daily_discounts,
AVG(aov) AS avg_aov,
SUM(unique_buyers) AS daily_buyers,
ROUND(
SUM(refund_count)::NUMERIC /
NULLIF(SUM(order_count), 0) * 100,
2
) AS refund_rate_pct
FROM mv_merchant_hourly_stats
GROUP BY merchant_id, DATE_TRUNC('day', hour_start);
You can also define alerting rules as materialized views that flag anomalies — for example, an unusual drop in GMV for a specific channel:
CREATE MATERIALIZED VIEW mv_gmv_anomalies AS
SELECT
merchant_id,
channel,
hour_start,
gmv,
LAG(gmv) OVER (
PARTITION BY merchant_id, channel
ORDER BY hour_start
) AS prev_hour_gmv
FROM mv_merchant_hourly_stats
WHERE gmv < LAG(gmv) OVER (
PARTITION BY merchant_id, channel
ORDER BY hour_start
) * 0.5; -- GMV dropped more than 50% vs previous hour
Step 4: Downstream / Serving
Sink the Gold-layer views to your application database or data warehouse so existing downstream consumers need minimal changes:
CREATE SINK sink_hourly_stats_to_postgres
FROM mv_merchant_hourly_stats
WITH (
connector = 'jdbc',
jdbc.url = 'jdbc:postgresql://app-db:5432/analytics',
table.name = 'merchant_hourly_stats',
type = 'upsert',
primary_key = 'merchant_id,channel,hour_start'
);
CREATE SINK sink_daily_summary_to_warehouse
FROM mv_merchant_daily_summary
WITH (
connector = 'jdbc',
jdbc.url = 'jdbc:postgresql://warehouse:5432/reporting',
table.name = 'merchant_daily_summary',
type = 'upsert',
primary_key = 'merchant_id,day'
);
Once these sinks are stable and downstream consumers are reading from the Postgres tables rather than the old batch outputs, you can decommission the batch jobs.
Architecture Comparison
| Dimension | Lambda Architecture | Kappa Architecture (RisingWave) |
| Pipelines | Batch + Streaming (two separate systems) | Single streaming SQL layer |
| Business logic | Defined twice | Defined once in SQL |
| Consistency | Divergence windows after each batch run | Continuously consistent |
| Backfill | Re-run batch jobs (hours) | Replay Kafka (minutes to hours, same code) |
| Latency | Minutes (batch) or seconds (stream) | Sub-second |
| Language | Java/Scala (Flink) + SQL (dbt/Spark) | SQL only |
| Cutover risk | — | Low — run in parallel before switching |
| Real-world outcome | Baseline | SHOPLINE: 76.7% API latency reduction |
FAQ
Q: What if my Kafka retention is limited and I can't replay full history? For initial backfill, you can use RisingWave's CDC connectors to seed materialized views from your existing database tables, then switch to Kafka consumption for the ongoing stream. This hybrid bootstrap pattern is common in production migrations.
Q: How do I handle schema evolution in Kafka topics?
RisingWave supports schema evolution via Confluent Schema Registry when using Avro or Protobuf encoding. For JSON topics, you can use ALTER SOURCE to add new columns as your event schemas change.
Q: Can I keep the data warehouse as a downstream consumer? Yes — and this is recommended. RisingWave handles the real-time serving path; your warehouse handles ad hoc analysis and long-term historical storage. Use the JDBC or Iceberg sink to feed the warehouse from RisingWave's Gold layer.
Key Takeaways
- Lambda architecture creates maintenance debt through duplicated pipelines — Kappa consolidates this into a single streaming SQL layer.
- RisingWave's incremental materialized views are the technical foundation: aggregations stay continuously up to date without recomputation.
- The migration strategy is risk-free: run RisingWave in parallel with your existing pipeline, validate outputs, then cut over.
- SHOPLINE's 76.7% API latency reduction demonstrates the practical impact of moving aggregation out of the query path.
- Start with your most painful consistency problem — usually GMV or order counts — and expand from there.
