The RisingWave team continues to deliver impressive features and updates with this new v1.4 release. This release continues to introduce new upstream and downstream connectors while improving upon existing ones. New functions and operators have also been added to allow for more complex data transformations. Follow along as we highlight some of the more notable features.

If you are interested in the full list of v1.4 updates, see the release note.

A new version of the S3 source connector

This release of RisingWave introduces a new version of the existing AWS S3 source connector. This improved version of the S3 source connector aims to resolve the scalability issues that the previous connector faced. The method of creating an S3 source with the improved connector is almost exactly the same as before. If you are interested in testing the integration between AWS S3 and RisingWave, we recommend using the upgraded version of the S3 source connector for a stable, hassle-free experience.

For more details:

See Ingest data from S3.
See the design document.

New sink connectors

The RisingWave team continues to improve RisingWave’s ability to connect to more systems, allowing RisingWave to be easily incorporated with any existing data stack. Check out our Integrations page for a list of supported systems. You can also vote to signal support for a specific integration to prioritize its development. With this release, RisingWave can sink data to Doris, Google BigQuery, and Redis. BigQuery, Doris, and Redis can all be used as a data storage tool. Using a simple SQL command, you can easily create a sink in RisingWave to output data to all these systems.

For more details:

See Doris sink documentation.
See Google BigQuery sink documentation.
See Redis sink documentation.
See Integrations page.

Support for COMMENT ON command

We now offer support for the COMMENT ON command. With this command, you can add comments to tables and columns, providing you with the ability to add documentation to the database objects. For current and future collaborators, the extra comments may allow them to quickly understand the purpose and usage of the tables and columns.

The following SQL query adds a comment to the table t1.

COMMENT ON TABLE t1 IS 'this is table 1'; 

To see the comment made, you can select from the rw_description RisingWave catalog.

SELECT * FROM rw_description; 
------- 
objoid | classoid | objsubid | description 
-------+----------+----------+------------------- 
  1001 |       41 |          | this is table 1 

You can also use the SHOW COLUMNS or DESCRIBE commands for the same purpose.

For more details:

See the documentation for the COMMENT ON command.

Support for subqueries in UPDATE and DELETE commands

You can now include a subquery when using the UPDATE or DELETE command. This allows for more complex data manipulations and easier data clean-up as you are able to perform updates and deletions based on specific conditions.

UPDATE table_name   SET col_1 = col_1 * 2.2 
  WHERE col_1 NOT IN (SELECT col_2 FROM drop_vals); 

For more details:

See the documentation for theUPDATE clause.
See the documentation for the DELETE clause.

A multitude of new JSON functions and operators

These new JSON functions and operators introduced in RisingWave handle JSONB types as inputs or outputs. In RisingWave, JSONB is a data type that enables you to create a column that stores JSON data. If being able to JSONB data types improves your data cleaning and transformation experience, these new features may come in handy.

This release introduces numerous JSON operators, allowing you to easily manipulate JSON data in RisingWave. Here are some examples of the new JSON operators:

The @> operator, which checks if the left JSONB object contains the values that are in the right JSONB object.

'[1, 2, 3]'::jsonb @> '[1, 3]'::jsonb → t

The - operator, which deletes a key-value pair, all matching keys, array elements, or an element at the specified index from a JSON object.

'{"a": "b", "c": "d"}'::jsonb - 'a' → {"c": "d"}

The #- operator, which deletes the element at the specified path. The path element can be field keys or array indexes.

'["a", {"b":1}]'::jsonb #- '{1,b}' → ["a", {}]

The full list of operators includes: @>, <@, ?, ?| , ?&, #>,  #>>, - and #-.

Four new JSON functions have also been added and they include jsonb_extract_path, jsonb_object, jsonb_pretty, and jsonb_strip_nulls. We will briefly introduce the jsonb_object and jsonb_pretty functions.

jsonb_object turns the input array of text elements and returns a JSONB object, where adjacent values in the string are read as key-value pairs.

jsonb_object(array['a', null]) → {"a": null}

jsonb_pretty outputs the input JSONB object as a nicely formatted text.

SELECT jsonb_pretty('[{"f1":1,"f2":null}, 2]');
------RESULT
[
    {
        "f1": 1,
        "f2": null
    },
    2
]

For more details:

See the documentation on JSON functions and operators.
See the JSONB data type.

One and a half year ago, in April 2022, we open sourced RisingWave, the distributed SQL streaming database. A quarter ago, in July 2023, we released the first official version of RisingWave, RisingWave 1.0, a battle-tested system that can be used in production. More recently, RisingWave 1.3 has been released

As an open-source streaming database released under Apache 2.0 license, the development team behind RisingWave actively collect feedback from users and strives to democratize stream processing: to make it simple, affordable, and accessible.

As a system that has been deployed in production in dozens of enterprises and fast-growing startups, how will RisingWave evolve? We plan to make it transparent and periodically update our roadmap. Here’s what you can anticipate in the future release of RisingWave.

Note that the roadmap is not final, and we will frequently update our roadmap to reflect and the item priority to better serve users.

Short-term goals (within the next 3 months)

Adaptive Scaling
Implement adaptive scaling to automatically adjust materialized view parallelism based on the number of CPU cores in the cluster.
Improvements on the Existing External Sinks
Optimize performance and improve stability of supported external sinks like Doris, Clickhouse, and Elasticsearch. We’ll also expand supported encoding formats for Kafka sink, including Protobuf, Avro, and the support for Schema Registry.
Iceberg Sink V2
We recently introduced a native integration with Iceberg, which is no longer based on the official Java library. It’s fully rewritten by Rust for performance and stability. We plan to stabilize it in the next few months.
Enhanced Observability
Expand system tables and add metrics for stateful operators to provide greater visibility into system health and performance.
Improved Open-source Web UI
Enhance RisingWave's open-source web UI with additional system information and monitoring capabilities.
Sink into table
Users may want to dynamically union the results of multiple views into a single table. For example, a view may correspond to an department in a company while there can be new departments once in a while. With this feature, users can seamlessly merging data from new views as they are added.
CDC Connection Sharing
RisingWave currently creates one CDC connection per table. Each connection will individually consume the replication logs, which consists of transactions not only to the source table, but also other tables in the same database. Therefore, multiple connections will lead to the duplicate consumption and a heavy load on the upstream database. Shared CDC connections can thus reduce the load and improve the stability of CDC.
Recoverable CREATE MATERIALIZED VIEW
Persist materialized view progress to allow recovering from failures without losing work already completed.
CDC Transaction Atomicity
CDC transactions in RisingWave currently applies by events, which may contain only partial content in a transaction. With the new feature, RisingWave will buffer all CDC events within a transaction until it can be fully applied atomically.
Parallel CDC Snapshot Loading
Introduce parallelism during CDC snapshot loading to improve user experience for large upstream tables.

Mid-term goals (within the next 6 months)

SSL/TLS Secured Connection
Implement SSL/TLS encryption for client/server communications to enhance security.
Alter Materialized View
Add ability to modify existing materialized views.
Session Window
Introduce session window functionality for advanced streaming analytics.
MemTable Spill
A refresh to a small table could suddenly cause 1k times amplification on write throughput. Such a case typically happens when there is a 10+ way join. A way to mitigate this is to use the local disk as a buffer for the flooded writes, thus avoiding OOM.
Dedicated Computes for Materialized View Creation
Some users complained that RisingWave’s materialized view creation is too slow, as it requires a resource-intensive ad-hoc computation. On the other hand, since the streaming (incremental computations) is long-running, it requires less resources at the same time. As a result, it’s possible to allocate dedicated resources for MV creation separately when needed, and deallocate them once finished.
More External Sinks
Redshift Sink and Snowflake Sink are in the plan.
Recursive CTE
Enable recursive common table expressions (CTE) to traverse hierarchical data like the organizational tree in a company.
Shared Meta Plane
Enable RisingWave clusters to share the meta plane, including Etcd (or Postgres in the future), to better utilize compute resources across clusters.

Long-term goals

Optimize analytical query performance on third-party systems like Presto and Trino
GraphQL API
To allow retrieving results from RisingWave directly through the browser.
Serverless Compaction
Automatically scale Compactor instances in and out to match workload demands in a serverless model.

We are thrilled to announce a strong partnership between RisingWave Cloud and Confluent Cloud, a leading figure for processing data in motion. RisingWave officially joins the Connect with Confluent program as a new verified partner. This marks a significant milestone in our journey to democratize stream processing.

As a company committed to staying at the forefront of innovation and ensuring that our clients have access to the best stream processing platform in the world, our decision to join the Confluent Partner Program was a natural choice.

What is RisingWave Cloud

RisingWave Cloud is a fully managed hosted service built to democratize stream processing. RisingWave Cloud offers a managed service for customers to create/drop/scale their RisingWave clusters, and build their stream processing applications easily with a few lines of SQL queries. Powered by RisingWave, RisingWave Cloud provides you with more flexibility, reliability, and scalability for your cloud-native streaming applications.

What is Confluent Cloud

Confluent Cloud is a fully managed streaming data service that harnesses the potential of Apache Kafka. Their services enable organizations to harness the power of real-time data, making it easier to build data-driven applications, gain valuable insights, and deliver exceptional customer experiences. To get start with Confluent Cloud, start your free trial here.

Key Benefits of Using Confluent Cloud with RisingWave Cloud

By partnering with Confluent Cloud, we can now leverage their cutting-edge technology and expertise to provide our customers with a complete set of solutions for capturing data in motion. Apache Kafka lies at the center of gathering and re-routing streaming data, however, it is still challenging to transform the streaming data on the fly to deliver real-time analytical insights, or to combine multiple streams with historical data for streaming ETL.

RisingWave provides a perfect match with Apache Kafka to deliver a collective solution for data in motion, from gathering data to real-time analytics. Whether you are looking to optimize your data pipelines, build event-driven applications, or enhance your data integration capabilities, our partnership with Confluent opens up a world of possibilities.

The combination of Confluent Cloud and RisingWave Cloud brings the following key advantages:

Managed Services: Both Confluent Cloud and RisingWave Cloud offer managed services with robust high availability and failure recovery solutions. On our side, RisingWave Cloud provides a very user-friendly experience so that customers can provision and manage their RisingWave clusters easily and quickly. This offloads the operational burden from your IT team, allowing them to focus on more strategic tasks.
Easy integration: With built-in Confluent Cloud source connectors, users can easily connect to an existing topic on a Confluent Cloud cluster from RisingWave Cloud. RisingWave Cloud will automatically deduce the schema from the schema registry service, and consume the data to deliver real-time insights.

Scalability and Cost-Efficiency: RisingWave Cloud can easily scale up/down/in/out your RisingWave clusters to accommodate changing workloads and storage needs. With our innovative compute-storage decoupled architecture, RisingWave Cloud achieves extreme cost-efficiency compared with other vendors.
Security: RisingWave Cloud guarantees a secure connection with Confluent Cloud only after being authenticated by API keys, which are managed by Confluent Cloud. RisingWave Cloud invests heavily in security measures under SOC2 compliance. No more worry about data leakage.
Solid and speedy technical support: Backed by RisingWave creators, our experts provide all necessary enterprise-grade technical support in any time zone.

This new release of RisingWave (v1.3) is packed with new and valuable features for sinks, sources, and SQL functions. Notable enhancements include advanced support for regular expressions, capabilities for delivering data to more downstream systems, and the addition of the WITH ORDINALITY clause.

For the full list of updates, see the release note.

Support for new array functions

With this release, multiple new array functions were added. These new functions include array_min, array_max, array_sort, array_sum. Array functions accept array types as inputs. These differ from aggregate functions, which perform calculations on a set of values. For more details on the new functions, including examples, look for the functions listed above on the Array functions page.

Advanced support for regular expression functions

We have introduced enhanced support for advanced features in our existing regular expression functions. These functions, including regexp_count, regexp_match, regexp_matches, and regexp_replace, now offer additional capabilities such as backreference, positive lookahead, negative lookahead, positive lookbehind, and negative lookbehind.

Support for WITH ORDINALITY clause

The WITH ORDINALITY clause is now also supported. When using set returning functions, this clause can be used at the end of the FROM clause in a SELECT statement. It will add a new column to the query results, which numbers the rows returned by the function, starting from 1. For more details, see WITH ORDINALITY.

SELECT * FROM
unnest(array[0,1,2])
WITH ORDINALITY;

Syntax update for creating Kafka, Kinesis, and Pulsar sinks

Previously, when creating a sink, the type of sink is specified under the WITH options with the type parameter. As an example, the following SQL query creates an append-only Kafka sink.

CREATE SINK s1 FROM mv WITH (
  connector = 'kafka',
  properties.bootstrap.server = 'localhost:9092',
  topic = 'test_topic',
  type = 'append-only');

Now the type of sink and data format can be specified with FORMAT...ENCODE... to be consistent with CREATE SOURCE. Note that the append-only type has been updated to PLAIN. For now, only the JSON format is supported. The following SQL query creates an append-only Kafka sink. Note that PLAIN is specified instead of append-only.

CREATE SINK snk FROM mv with (
  connector = 'kafka',
  properties.bootstrap.server = 'localhost:9092',
  topic = 'test_topic')
FORMAT PLAIN ENCODE JSON;

This syntax change only applied to Kafka, Kinesis, and Pulsar sinks. For more details, see the Sink data to Kafka page or the Sink data to AWS Kinesis page.

A group of new sink connectors

This release introduces support for multiple new sink connectors, allowing you to sink data transformed in RisingWave into more systems. RisingWave strives to provide users the ability to deliver data to popular downstream systems. Check out our Integrations page for a list of supported systems. You can also vote to signal support for a specific integration to prioritize its development.

Cassandra and ScyllaDB sink

Cassandra and ScyllaDB are open-source NoSQL database management systems capable of handling large workloads. Since ScyllaDB is built as a replacement for Cassandra, the RisingWave sink connector for both are the same. See the Cassandra sink guide for more details.

Doris sink

Apache Doris is an open-source distributed analytical database designed for ad-hoc analysis of large volumes of data. See how simple the process is to deliver data from RisingWave to Doris with this guide.

Elasticsearch sink

Elasticsearch is an open-source distributed search and analytics engine designed to process and handle large values of data in real time. Follow our guide to see how to create an Elasticsearch sink in RisingWave.

NATS messaging system

NATS is an open-source messaging system, which follows a pub-sub and request-response messaging pattern, ideal for real-time applications. See the documentation for details on how to sink data to NATS.

Pulsar sink

Apache Pulsar is an open-source distributed messaging system, designed for developing real-time, event-driven applications. With RisingWave, you can now ingest data from and sink data to Pulsar. See our new guide on how to create a sink to deliver data to Pulsar.

We are thrilled to announce the release of RisingWave v1.2. Here are the key highlights in this release.

Emit-on-window-close syntax change

The syntax for the emit-on-window-close policy has changed. If your application uses the old syntax, please see the updated documentation and update your code accordingly to avoid any issues.

For more details about Emit-on-window-close, see our documentation.

Streaming job management

With the introduction of the CANCEL JOBS and SHOW JOBS SQL queries, you can now cancel or see the progress of streaming jobs. In RisingWave, a streaming job creates a materialized view. This allows you to save time by canceling streaming jobs that are no longer necessary.

For more details about streaming job management, see our documentation.

Lateral subqueries

This release supports lateral subqueries, which provide you with more methods to perform complex calculations and efficient data transformations. The keyword LATERAL can be included before a subquery’s SELECT statement, enabling you to reference columns from preceding subqueries.

For more details about lateral subqueries, see our documentation.

Sink data to ClickHouse

We now support the integration of the popular database management system ClickHouse. With a straightforward CREATE SINK query, you can easily sink data from RisingWave to ClickHouse.

For more details about sink data to ClickHouse, see our documentation.

New version of Iceberg sink connector

BREAKING CHANGE: The Iceberg sink connector has been updated. If your application included sinking to Iceberg, please review the syntax and parameters on the Sink data to Iceberg guide to ensure everything continues to run smoothly.

For more details about new version of Iceberg sink connector, see our documentation.

Transactions within a CDC table

When you use a native RisingWave sink connector, such as MySQL, PostgreSQL, or Citus CDC, you can enable transactions within a CDC table.

For more details about how to enable this feature, see our documentation.

LIKE expressions

In this release, we support [NOT] LIKE and [NOT] ILIKE pattern-matching expressions, as well as the respective [!]~~ and [!]~~* operators. LIKE expressions can be used in SHOW SQL queries, giving you the option to filter the output for relevant results.

For more details about LIKE expressions, see our documentation.

RisingWave Cloud: next-gen stream processing

Powered by RisingWave, RisingWave Cloud offers a managed service that provides you more flexibility, reliability and scalability for your cloud-native streaming applications.

In particular, RisingWave Cloud holds the following key advantages:

Managed Services: RisingWave Cloud offer managed RisingWave services that offer robust high availability and failure recovery solutions. RisingWave cloud will automatically handle routine maintenance tasks such as backups, patch management, and security updates. This offloads the operational burden from your IT team, allowing them to focus on more strategic tasks.
Scalability and Cost-Efficiency: RisingWave Cloud can easily scale up/down/in/out your RisingWave clusters to accommodate changing workloads and storage needs. With our innovative compute-storage decoupled architecture, RisingWave Cloud achieves extreme cost-efficiency comparing with other vendors.
Security: RisingWave Cloud invest heavily in security measures under SOC2 compliance. No more worry about data leakage.
Rapid Deployment: RisingWave Cloud provides very user-friendly experience so that customers can provision and manage their RisingWave clusters easily and quickly. The package of builtin monitoring tools, web query editors and source/sink integrations putting together delivers an one-stop development solution.
Solid and speedy technical support: Backed by RisingWave creators, our experts provide all necessary enterprise-grade technical supports in any time zone.

RisingWave Cloud on AWS Marketplace

We are thrilled to announce that RisingWave Cloud has landed on AWS Marketplace! The AWS Marketplace integration offers a streamlined and simplified process of discovering, procuring, and deploying RisingWave on the Amazon Web Services (AWS) cloud platform.

With RisingWave Cloud on AWS Marketplace, you get seamless integration with the industry-leading reliable infrastructure of AWS and its ecosystem. It simplifies the entire software lifecycle, from selection to deployment to ongoing management, making it a valuable resource for organizations looking to leverage the full potential of AWS services and third-party software offerings. Get started now！

We are thrilled to announce the new release of RisingWave: the Rust-written, open-source streaming database. RisingWave v1.1 is packed with many exciting new features. Here are the key highlights.

Watermarks

Real-time data ingestion is inherently unordered. To properly handle out-of-order events, we need watermarks.

Accordingly we introduce watermarks in this release. In RisingWave, you can define watermarks in sources and append-only tables. Once you define them, you can use them in time-window aggregations and other stateful operations.

Append-only tables is also a new feature that was introduced in this version to facilitate the definition of watermarks.

For details about watermarks, see Watermarks.

Emit-on-window-close policy

For windowed calculations, we offer the flexibility to emit results either upon update or when the window closes.

Emitting results upon window close is particularly advantageous in certain situations.For example, when writing to append-only downstream systems like Kafka or S3, or when dealing with calculations that benefit from triggering only when the window closes, such as percentile calculations.

Our emit-on-window-close policy, introduced in v1.1, caters to these scenarios.

For details about this policy, see Emit on window close.

Read-only transactions

We recognize the significance of transactions in ensuring data processing consistency and reliability.

In this release, so we have incorporated support for read-only transactions. All reads within a transaction are executed against the consistent snapshot, guaranteeing data consistency as they originate from the same point-in-time.

For details about Read-only transactions, see Transactions.

New window functions

To enhance your data analysis capabilities, we have introduced several new window functions in RisingWave v1.1. These include lead(), lag(), first_value(), and last_value().

For details about new window functions, see Window functions.