In this installment, we would like to explain the rationale behind the need for a unified data processing framework and how RisingWave’s streaming-first approach is pioneering this vision.
This blog post continues our series of thought leadership pieces that provide insights into the product strategy and evolution for RisingWave (see the first installment here). In this installment, I would like to explain the rationale behind the need for a unified data processing framework and how RisingWave’s streaming-first approach is pioneering this vision.
As we unveil the release of RisingWave 2.0 GA, we want to take a moment to reflect on the broader data management landscape. Stream processing has evolved from a trend to a critical necessity for innovative data-driven organizations. Historically, batch and streaming workloads were treated separately, requiring distinct data infrastructures. However, leading data platform vendors now recognize the importance of converging batch and stream processing.
As business challenges have evolved, the line between batch and streaming has blurred, paving the way for a unified data processing model.
Incumbent batch-oriented systems have attempted to handle streaming workloads through “micro-batching,” essentially a faster version of traditional batch processing. However, processing entire tables whenever new data arrives is inefficient and costly, especially as use cases become more complex and diverse, resulting in slower results and increased expenses.
In contrast, streaming systems offer superior efficiency and performance by processing incremental changes rather than relying on full micro-batched or snapshotted tables. This approach enables users to write queries that combine and layer derivative and intermediate data, ensuring real-time updates.
What’s truly needed is a shift in perspective, where workloads are organized based on data latency needs rather than focusing solely on query performance. This model places workloads on a spectrum that prioritizes data latency requirements, aligning processing approaches with the specific needs of the data.
Let’s take a closer look at the concepts of data latency and query latency. Data latency refers to the time between when data is generated and when it is ready for processing and analysis. This latency is typically influenced by factors like network speed and ingest overhead. In contrast, query latency is the delay between when a query is submitted to a data processing system and when the results are returned. This type of latency is mainly affected by query complexity, data volume, storage type, and the sophistication of the query engine.
The term “real-time” is frequently overused in the data industry, with no vendor openly marketing their platform as a ‘non-real-time’ solution. However, understanding the key technical and functional differences can help practitioners see beyond the marketing hype.
For streaming databases like RisingWave, “real-time” means generating insights from live data, with systems optimized for processing and analyzing high-volume, high-velocity data nearly instantaneously. In contrast, traditional batch systems use “real-time” to describe the delivery of query results, focusing on processing large volumes of historical data in batch mode. While batch systems are effective for identifying past trends and patterns, they fall short in supporting in real-time decision-making or embedding analytics into downstream applications.
Ultimately, modern businesses are looking for more than just historical insights or a snapshot of current events. They now prioritize advanced analytics that blend live data with historical information to deliver deeper, actionable insights. RisingWave 2.0 marks the beginning of our journey to address this new frontier and meet the evolving needs of these use cases.
Traditional data systems built on batch processing are poorly equipped to handle workloads with varying data latency requirements. These systems lack native streaming connections to data sources and sinks, relying instead on fragile ETL processes and complex data pipelines. As a result, users must maintain data consistency through intermediate data sets, which significantly increases data latency and adds substantial operational overhead.
RisingWave’s streaming-first approach effectively addresses both data latency and query latency requirements, ensuring optimal performance across a wide range of use cases. Now, let’s explore the key features that enable RisingWave to provide a unified solution for both streaming and batch workloads.
RisingWave began its journey as a stream processing system. The design choices made during the development of RisingWave 1.0 laid a solid foundation for addressing a wide range of real-time data workloads. With RisingWave 2.0, we are building on this foundation by introducing features that advance our vision of a unified data processing model. As I conclude this blog, it’s important to acknowledge that while the goal of unified data processing is commendable, it may seem overly ambitious. However, we believe that “unified” doesn’t necessarily imply a single platform that addresses every type of workload comprehensively. Instead, RisingWave is uniquely positioned to meet this challenge with our streaming-first approach, focusing on delivering efficient and flexible solutions for modern data needs.
