Streaming Database

A Streaming Database is a specialized type of database system designed to ingest, store, process, and serve data from continuous, real-time data streams. Unlike traditional databases that are optimized for storing and querying static data-at-rest, streaming databases are built to handle data-in-motion, enabling continuous query execution, incremental state management, and low-latency access to fresh, dynamically updating results.

RisingWave is an example of a distributed SQL streaming database.

Core Capabilities and Characteristics

Continuous Data Ingestion: Natively designed to connect to streaming sources (like Kafka, Pulsar, Kinesis, CDC streams) and continuously ingest unbounded data.
Stream Processing Engine: Incorporates a powerful stream processing engine capable of performing complex operations on incoming data, including:
- Filtering and transformations
- Stateful operations like joins (stream-stream, stream-table)
- Windowed aggregations
Continuous Queries: Allows users to define queries (often using SQL or a similar declarative language) that run continuously over the data streams, producing results that update in real-time as new data arrives.
Incremental Computation / Materialized Views: A key feature of many streaming databases (and central to RisingWave) is the ability to incrementally update the results of continuous queries, often by storing them in materialized views. This means only the changes in input data are processed to update the results, rather than recomputing everything from scratch, leading to very low latency.
State Management: Provides robust and scalable mechanisms for managing the state required for stateful stream processing operations (e.g., join states, aggregation states). This state is typically persisted for fault tolerance.
Low-Latency Querying of Results: Enables fast querying of the (often materialized) results of streaming computations, allowing applications to access fresh insights immediately.
Data Serving: Can serve as a source of truth for real-time applications, dashboards, and operational systems that need up-to-the-millisecond data.
Connectors: Includes a rich set of connectors for both ingesting data from various sources and sinking results to external systems.
SQL Interface (Common): Many modern streaming databases, including RisingWave, offer a SQL interface, making stream processing accessible to a wider audience familiar with database query languages. This contrasts with earlier stream processing systems that often required programming in languages like Java or Scala.

Why Use a Streaming Database?

Simplified Real-time Architectures: Can replace complex "Lambda" or "Kappa" architectures that traditionally combined separate batch and streaming systems. A streaming database can often handle both real-time processing and serving of fresh data.
Freshness and Low Latency: Provides much fresher data and lower end-to-end latency compared to batch-oriented systems.
Developer Productivity: Using SQL or a high-level declarative language can significantly speed up the development of real-time applications.
Consistency: Can provide stronger consistency guarantees for streaming results compared to ad-hoc combinations of different tools.
Operational Efficiency: Consolidates multiple functions (ingestion, processing, storage, serving) into a single system, potentially reducing operational overhead.