Subscription

In RisingWave, a Subscription is a feature that allows external applications or services to consume changes from a Materialized View or a Table in a streaming fashion. When data in the source Materialized View or Table changes (due to incoming stream data being processed), RisingWave automatically sends these changes (inserts, updates, deletes) to active subscribers.

This provides a powerful mechanism for downstream systems to react to real-time updates and maintain consistency with the data managed by RisingWave, without needing to repeatedly poll the views or tables for changes.

Key Characteristics

1. Change Data Capture (CDC) for Materialized Views/Tables: Subscriptions effectively provide a CDC stream from RisingWave's internal state.
2. Push-based Model: RisingWave pushes changes to subscribers, rather than subscribers pulling data.
3. Real-time Updates: Subscribers receive updates with low latency as soon as the underlying materialized view or table is updated.
4. Decoupling: Decouples downstream applications from the internal workings of RisingWave's data processing. Downstream systems only need to understand how to consume the change stream.
5. Simplified Integration: Makes it easier to integrate RisingWave with various downstream systems, such as:
   • External databases or data warehouses for further archiving or batch processing.
   • Caching layers.
   • Real-time alerting or notification systems.
   • Microservices that need to react to data changes.
   • Real-time dashboards or applications that need to display the latest data.

How it Works (Conceptual)

1. Define a Materialized View or Table: You first have one or more Materialized Views or Tables in RisingWave that are continuously updated by streaming queries.
2. Create a Subscription: You define a subscription on a specific Materialized View or Table.

   -- Example (syntax might vary based on the exact RisingWave version and target)
   CREATE SUBSCRIPTION my_subscription
   FROM my_materialized_view
   PUBLISH VIA KAFKA 'kafka_broker_address' TOPIC 'my_topic'
   FORMAT JSON;
   

3. Specify Publication Details: When creating a subscription, you typically specify:
   • The source Materialized View or Table.
   • The sink or transport mechanism (e.g., Kafka, Webhook).
   • The format of the change data (e.g., JSON, Avro, Debezium JSON).
   • Connection details for the sink (e.g., Kafka broker address, topic name).
4. Change Detection: RisingWave monitors the source Materialized View or Table for any changes (new rows, updated rows, deleted rows).
5. Data Emission: When changes are detected, RisingWave formats them according to the subscription's definition and sends them to the specified sink.
   • For example, if publishing to Kafka, RisingWave would produce messages to the designated Kafka topic. Each message would represent a change event.
6. Consumption by Subscribers: External applications or services consume these change events from the sink (e.g., by consuming from the Kafka topic).

Use Cases

• Feeding Real-time Dashboards: Push updates directly to dashboarding tools or applications.
• Triggering Downstream Workflows: Initiate actions in other microservices or systems based on data changes.
• Populating External Databases/Caches: Keep external data stores or caches synchronized with RisingWave's materialized views.
• Real-time Alerting: Send notifications when specific conditions are met in the data.
• Inter-service Communication: Facilitate event-driven communication between services based on data state changes.
• Data Replication/Distribution: Distribute processed data to other systems or regions.

Benefits in RisingWave

• Efficiency: Avoids the need for downstream systems to constantly query RisingWave, reducing load on the database.
• Low Latency: Provides near real-time propagation of changes.
• Consistency: Helps maintain data consistency between RisingWave and downstream systems.
• Simplified Application Logic: Downstream applications can be simpler as they just react to incoming change events.

Subscriptions are a powerful feature in RisingWave for building reactive, event-driven architectures and integrating processed streaming data with the wider data ecosystem.