TL;DR
RisingWave 3.1 makes real-time data processing more flexible, efficient, and lakehouse-ready. This release introduces several key enhancements:
Native VARIANT Support: Process rich semi-structured data with preserved type information, improving interoperability with Apache Iceberg, Snowflake, and Doris.
Enhanced Apache Iceberg Integration: More efficient upserts for Iceberg V2/V3, smarter compaction, automatic manifest maintenance, and improved metadata visibility.
Periodically Refreshed Materialized Views: Balance data freshness and compute efficiency by refreshing materialized views at configurable intervals instead of continuously.
Flexible Sink Management: Replace sinks without full historical backfills, selectively process new changes, and replay retained data from specific timestamps.
Performance and Reliability Improvements: Better locality backfill, granular rate limiting, expanded SQL capabilities, improved observability, and more robust connectors.
Bottom line: RisingWave 3.1 further unifies streaming processing, incremental computation, and analytical querying, helping developers build and operate real-time data pipelines with less complexity and greater efficiency.
This release focuses on four key areas: semi-structured data, Apache Iceberg integration, flexible computation, and simplified pipeline management.
1. Native VARIANT Support for Semi-Structured Data
RisingWave 3.1 introduces VARIANT as a first-class SQL data type, enabling richer support for flexible and evolving data structures.
Unlike JSONB, VARIANT preserves native types such as decimals, timestamps, binary values, arrays, and nested structures. Built on Apache Parquet Variant V1, it improves interoperability with modern data platforms, including Snowflake, Apache Iceberg, and Doris.
New SQL functions make VARIANT values easier to work with:
to_variant()– Convert SQL values to VARIANT.variant_get()andtry_variant_get()– Extract typed values.variant_typeof()– Inspect underlying data types.
RisingWave also supports writing JSONB values to Doris VARIANT columns and improves error handling for malformed VARIANT data.
Why it matters: Developers can process and exchange complex, semi-structured data without sacrificing type information, making RisingWave better suited for IoT, AI, and lakehouse workloads.
2. Apache Iceberg: More Efficient Upserts and Smarter Compaction
RisingWave 3.1 significantly expands its native Apache Iceberg capabilities, strengthening its role as a streaming engine for open lakehouse architectures.
Key improvements include:
Primary-key-indexed upserts: Iceberg V2 and V3 merge-on-read sinks now support enable_pk_index, enabling more efficient inserts, updates, and deletes using position deletes and deletion vectors.
Native VARIANT reads: RisingWave can read Iceberg V3 VARIANT columns through Iceberg sources and refreshable Iceberg tables.
SQL-based metadata visibility: New metadata relations—$snapshots, $manifests, and $files—provide direct visibility into Iceberg snapshots, manifests, and data files.
Smarter compaction: The new compaction.type = 'auto' option for merge-on-read sinks selectively compacts files based on factors such as file size and delete density, reducing unnecessary rewrites.
Automatic manifest maintenance: Newly created Iceberg Engine V2 tables automatically benefit from manifest rewriting to reduce metadata overhead.
These enhancements build on RisingWave's existing Rust-based Iceberg connectors, Iceberg Table Engine, exactly-once delivery, and Apache DataFusion-powered batch query engine.
Why it matters: Organizations can maintain continuously updated Iceberg datasets with greater efficiency, lower maintenance overhead, and better operational visibility.
3. Periodically Refreshed Materialized Views
Not every workload requires continuous updates. Some analytical queries only need fresh results every few minutes or hours.
RisingWave 3.1 introduces periodically refreshed materialized views, providing an alternative to continuously maintained materialized views.
Instead of processing every incoming change, these views periodically recompute results from consistent upstream snapshots.
The refresh interval is configurable through refresh.interval.sec.
This capability is particularly useful for:
Analytical dashboards and reporting.
Expensive aggregations over large datasets.
Workloads where periodic freshness is sufficient.
By combining continuous streaming computation with snapshot-based refreshes, RisingWave gives developers greater control over the trade-off between data freshness and compute overhead.
Why it matters: Applications can choose the most appropriate computation model without introducing separate batch processing infrastructure.
4. More Flexible Sink Management
RisingWave 3.1 makes it easier to modify and operate production streaming pipelines without unnecessary historical replay or costly rebuilds.
Replace sinks without rebuilding pipelines: The new REPLACE SINK ... FROM ... command allows existing sinks to switch upstream sources at a coordinated cutover point without a full historical backfill, subject to current connector and syntax limitations.
Enhanced snapshot-free sinks: Sinks configured with snapshot = false now support projections and filters, allowing applications to process only new changes while selectively transforming data.
Timestamp-based replay: The new since_timestamp option allows sinks to replay retained changelog data from a specified timestamp rather than processing the entire historical dataset.
Additional improvements include dynamic HTTP sink URLs, Snowflake serverless task support, and configurable turbopuffer namespace sharding.
Why it matters: Developers can evolve streaming pipelines with less disruption, faster initialization, and greater control over historical data processing.
5. Additional SQL, Performance, and Reliability Improvements
RisingWave 3.1 also delivers enhancements across the platform:
SQL: Partition-aware
GAP_FILL,clock_timestamp(), andgen_random_uuid().Performance: Automatic locality backfill selection and more granular rate limiting for ingestion, backfill, DML, and sinks.
Observability: Improved barrier metrics, actor traces, and fragment-level diagnostics.
Connectors: Enhanced PostgreSQL CDC, Google Pub/Sub, Pulsar, and StarRocks integration.
Reliability: Fixes for CDC processing, streaming recovery, temporal joins, and sink operations.
Together, these improvements make RisingWave more efficient, observable, and reliable for demanding production workloads.
Looking Ahead
RisingWave 3.1 continues our journey toward a unified platform for streaming data processing, incremental computation, and analytical querying.
With native VARIANT support, deeper Apache Iceberg integration, flexible materialized views, and improved sink management, developers have more options to build efficient real-time applications without unnecessary architectural complexity.
Building on RisingWave 3.0's integration of Apache DataFusion as the default batch execution engine, this release further bridges the gap between streaming and batch analytics.
Our vision remains simple: One platform, familiar SQL, and fresh data wherever your applications need it.
