Hummock (RisingWave Specific)
Hummock is the cloud-native, Log-Structured Merge-tree (LSM-tree) based storage engine specifically designed and built for RisingWave. It serves as the distributed state store that underpins RisingWave's ability to manage large volumes of streaming state efficiently and durably. Hummock persists state to cloud object storage (like AWS S3, Google Cloud Storage, or Azure Blob Storage), enabling the separation of storage and compute, which is a key architectural principle of RisingWave.
Core Purpose and Functionality
The primary role of Hummock is to provide:
- Durable State Persistence: Ensuring that the state of streaming computations (e.g., aggregations, joins, windowed results) is safely stored and can survive failures.
- Efficient State Access: Providing low-latency read and write access to state for the compute nodes executing streaming dataflows.
- Scalable State Management: Handling terabytes of state as data volumes and computation complexity grow.
- Consistent Snapshots (Checkpointing): Supporting RisingWave's checkpointing mechanism by creating consistent versions of the entire state, allowing for fault tolerance and recovery.
- Versioning and Time Travel (Future Potential): The underlying LSM-tree structure with versioned data facilitates features like querying historical state, although this is more of a future roadmap item for point-in-time queries on state.
Key Architectural Features
- LSM-Tree Based: Hummock uses an LSM-tree architecture. This means incoming writes are first buffered in memory (mem-tables) and then flushed to immutable, sorted runs on persistent storage (SSTables - Sorted String Tables). These SSTables are periodically compacted in the background to merge data, remove obsolete entries, and optimize read performance. This architecture is well-suited for write-intensive workloads and large datasets.
- Cloud-Native Design:
- Separation of Storage and Compute: Compute nodes in RisingWave are relatively stateless regarding the core processing logic. They fetch and store state from/to Hummock, which runs as a separate layer interacting with cloud object storage. This allows independent scaling of compute and storage resources.
- Leverages Object Storage: Hummock uses highly available, durable, and scalable cloud object storage as its primary backing store for SSTables. This offloads the burden of data replication and durability to the cloud provider.
- Tiered Storage:
- MemTable (Memory): For fast writes and reads of recent data.
- SSTables on Object Storage: For durable, long-term storage of versioned state.
- Block Cache (Compute Nodes): Compute nodes maintain a local cache of frequently accessed data blocks from SSTables to reduce latency for reads from object storage.
- Centralized Metadata Management: The RisingWave Meta Node manages Hummock's metadata, including the structure of SSTables, version information, and compaction status.
- Compaction Service: Dedicated Compactor Nodes (or a compaction service within Meta Nodes in some configurations) perform the background compaction of SSTables. This is crucial for maintaining read performance and managing storage space.
- Snapshot Isolation for Checkpoints: Hummock provides consistent snapshots of the state at checkpoint boundaries. This ensures that if a failure occurs, RisingWave can roll back to a globally consistent state.
How Hummock Interacts with RisingWave Components
- Compute Nodes: When processing data, compute nodes read the necessary state from Hummock and write updated state back. They interact with Hummock via a client library.
- Meta Node: The Meta Node orchestrates checkpointing. It instructs compute nodes to flush their state and coordinates with Hummock to commit a new version (snapshot) of the state. The Meta Node also manages Hummock's metadata and assigns compaction tasks.
- Compactor Nodes: These nodes retrieve SSTables from object storage, perform compaction, and write new, compacted SSTables back, updating the metadata via the Meta Node.
Benefits of Hummock
- Scalability: Can handle very large state sizes due to its LSM-tree architecture and use of cloud object storage.
- Elasticity: The separation of compute and storage allows for flexible scaling. Compute resources can be scaled up or down based on load without affecting the stored state.
- Cost-Effectiveness: Leveraging cheap and scalable cloud object storage.
- Fault Tolerance & Durability: Relies on the high durability of cloud object storage and RisingWave's checkpointing mechanism for recovery.
- Performance: Optimized for streaming workloads with techniques like in-memory buffering, block caching, and background compaction.
Related Glossary Terms
