Unlocking the Power of Data with RisingWave's Enhanced Iceberg Connector

What’s Apache Iceberg

Apache Iceberg is a widely adopted open table format that plays a critical role in the lakehouse architecture. As a highly popular table format, Apache Iceberg offers several key features:

Multi-engine support. The Apache Iceberg specification is not bound to any specific compute engine or programming language. It provides a clear definition that can be implemented across different systems.
High performance. Apache Iceberg is designed to handle large volumes of data efficiently.
ACID guarantees. Apache Iceberg ensures Atomicity, Consistency, Isolation, and Durability (ACID) properties for data operations.

There are also advanced features such as branching and time travel that users can explore further.

Why we implemented a new version of the connector

We introduced a new version of the Iceberg sink connector in RisingWave to address various issues encountered with the original implementation. The original version wrapped the Java library in Rust to meet user demand, but we have identified several issues as more people used it.

The issues included performance problems caused by serialization/deserialization and unnecessary memory allocation during the conversion of data to Java records. Additionally, ensuring compatibility with Iceberg's advanced features, like multi-catalog support and hidden partitioning, proved challenging due to the cross-language conversion.

To overcome these issues and better serve our users, we decided to rewrite the Iceberg connector using Rust. The implementation process was smooth, thanks to the well-defined specification and design of Apache Iceberg. We have now contributed the new version to the apache/iceberg-rust repository. The documentation for the new version of the connector can be found here.

The architecture

One of the distinguishing features of Apache Iceberg is its clear separation of the metadata layer from the data layer. This design allows compute engines to scale their read and write processes independently while maintaining ACID (Atomicity, Consistency, Isolation, Durability) guarantees on limitless storage systems such as S3 or HDFS.

The Iceberg sink connector in RisingWave follows a design that separates the process of writing data from the act of committing it to the Iceberg catalog. When a checkpoint is triggered, we flush the data into Parquet files and then proceed to commit the transaction to the Iceberg catalog. This approach allows for scalable data ingestion while still upholding the ACID guarantees provided by the Iceberg table.

How we leverage streaming update support in Iceberg v2

In early days, Iceberg was designed for append-only workloads, and Iceberg v2 has added support for deletions, which enables efficient streaming updates. Apache Iceberg features two types of deletions: equation deletion and position deletion.

Equation deletion: used to delete data with same equation deletion key in previous versions.
Position deletion: used to delete data with same position in previous or current versions.

In the new version of the Iceberg connector, the process of updating the count of Jerry follows the following steps:

Initially, when executing the action update count = 5 where name = ‘Jerry’ , RisingWave creates an equation delete entry to remove rows with the condition name = 'Jerry'. Following that, it writes the data ('Jerry', 5) into the data file.
At this point, we have recorded that the location of the data corresponding to name = 'Jerry' is in the file s3://test/1.parquet at position 3.
When executing the action update count = 7 where name = ‘Jerry’, RisingWave creates a position delete entry ('s3://test/1.parquet', 3) to remove the previous row. Following that, it writes the data (’Jerry’, 7) into the data file.

Possible table maintenance issues and workarounds

The RisingWave Iceberg connector offers high-throughput ingestion. However, there are some common issues that can arise:

Excessive snapshots: Committing to Iceberg tables creates new snapshots, potentially increasing metadata file size.
Numerous small files: Frequent commits to Iceberg tables can generate small files, which can negatively impact read performance.
Abundance of deletion files: While Iceberg + RisingWave enables efficient updates, having many deletion files can also affect read performance.

The Iceberg community has provided several procedures to mitigate these problems:

Use the expire_snapshots and remove_orphan_files procedures to retire old snapshots and free up data files.
Utilize the rewrite_position_delete_files procedure to consolidate small position delete files into larger ones.
Employ the rewrite_data_files procedure to completely remove deletion files, resulting in optimal read performance.