The Rise of Lambda Architecture — Balancing Real-Time Speed and Batch Accuracy

In the previous article, we discussed how synchronous processing leads to coupling and scalability challenges as traffic grows. Naturally, one might think: “Why not just use streaming to process everything?”

However, in the early 2010s, streaming-only architectures faced significant technical hurdles.

Back in the early days of big data, real-time processing was out of reach. Hadoop and MapReduce excelled at batch processing massive datasets, but they were almost powerless when it came to second-level or minute-level real-time queries. Lambda Architecture was born to address this exact pain point—a classic solution that balanced latency and accuracy.

Lambda Architecture design principles

Lambda Architecture adopts a layered processing strategy, dividing data processing into three independent layers:

                    Raw Data Stream
                          │
                ┌───────────────────┐
                │                   │
                ▼                   ▼
        ┌─────────────┐      ┌──────────────┐
        │ Batch Layer │      │ Speed Layer  │
        │             │      │              │
        │             │      │              │
        └─────────────┘      └──────────────┘
                │                    │
                │                    │
                │                    │
                └──────────┼─────────┘
                           ▼
                ┌─────────────────────┐
                │   Serving Layer     │
                └─────────────────────┘

Three layers explained

Batch layer

• Role: Processes full datasets to ensure eventual correctness

• Characteristics: Slow processing but fully accurate; ideal for complex analytics

• Tech stack: Hadoop, Spark, Hive

Speed layer

• Role: Processes newly arrived data, providing low-latency results

• Tech stack: Storm, Flink, Kafka Streams

Serving layer

• Role: Merges results from Batch and Speed layers and provides a unified query interface

• Characteristics: Users don’t care where the data comes from—they just get accurate, near-real-time results

• Tech stack: Cassandra, HBase, Elasticsearch

The core value of this design is achieving both real-time insights and eventual accuracy.

How it works in practice

In typical implementations of Lambda Architecture :

Batch layer processing flow:

• Use Hadoop, Hive, or Spark to process historical data

• Recompute the complete dataset periodically (e.g., daily or every few hours)

• Write results to the Serving DB

Speed layer processing flow:

• Use Storm, Spark Streaming, or Flink to process latest data

• Only process recent data windows (seconds to hours)

• Update the same Serving DB in real time

Query integration:

Clients only query the Serving DB; the Batch and Speed layer results are seamlessly merged behind the scenes.

Lambda Architecture in code

💡Note

All code below is pseudo code meant to illustrate concepts and workflows.

Example: real-time e-commerce order statistics

from flask import Flask, jsonify

app = Flask(__name__)

@app.route("/query/<merchant_id>")
def query(merchant_id):
    batch_result = query_batch_table(merchant_id)  # Batch layer result
    speed_result = query_speed_table(merchant_id)  # Speed layer result
    combined_result = batch_result + speed_result   # Merge results
    return jsonify({"merchant_id": merchant_id, "order_count": combined_result})

def query_batch_table(merchant_id):
    # Simulate Batch layer query (full historical data)
    return 1000  # Example: 1000 orders from batch

def query_speed_table(merchant_id):
    # Simulate Speed layer query (today's incremental data)
    return 50  # Example: 50 new orders today

if __name__ == "__main__":
    app.run(debug=True)

This snippet demonstrates the essence of Lambda Architecture: dynamically merging Batch and Speed results at query time, giving users near-real-time yet ultimately accurate data.

Pros and cons

Pros

1. Overcomes infrastructure limitations: Speed Layer compensates for Hadoop’s high latency

2. Clear and scalable architecture: Batch and Speed layers can scale independently

3. Simplified query layer: Clients only interact with one Serving DB

4. Fault-tolerant: Two systems provide mutual backup

Cons

1. High development and maintenance cost: Must maintain two processing logics and keep them consistent

2. Complex debugging: Dual processing paths make data tracing harder

3. High demands on Serving DB: Must handle both high-concurrency writes and stable queries

Conclusion

Lambda Architecture was a game-changer in the Hadoop era, allowing companies to achieve near-real-time analytics for the first time. Although it doubled the development and maintenance workload, its evolution shows that technical architectures are often the best compromise given current capabilities.

Over the past three articles, we explored:

• Why the HTTP synchronous model is commonly used

• The challenges of synchronous processing and the need for asynchronous alternatives

• How Lambda Architecture balances latency and accuracy

Next, it’s time to move from theory to practice.

Starting from Day 4, we’ll move into hands-on practice, beginning with a basic Python implementation of a Kafka consumer. Using simple pseudo code, we’ll gradually explore and understand the core functionalities and operations behind a Stream Processing Engine.

Ready to get your hands dirty with some coding? Let’s see how engineers back then built powerful streaming systems using the simplest tools available.

About the author

Shih-min (Stan) Hsu — Team Lead, Data Engineering at SHOPLINE | Expert in Batch and Streaming Data Pipelines.