Apache Iceberg Partition Evolution: Change Partitions Without Rewriting Data
Partition evolution lets you change the partitioning scheme of an Iceberg table without rewriting existing data. This is unique to Iceberg — Delta Lake and Hudi require full table rewrites to change partitioning. For streaming workloads where data volumes grow over time, partition evolution is critical.
How It Works
-- Table originally partitioned by day
ALTER TABLE events ADD PARTITION FIELD hours(event_time);
-- New data partitioned by hour; old data retains daily partitioning
New data uses the new partitioning scheme. Old data stays as-is. Queries automatically handle both schemes.
Why It Matters for Streaming
As streaming tables grow:
- Day 1: 10K events/day → daily partitions are fine
- Month 6: 1M events/day → daily partitions are too large, need hourly
- Year 2: 10M events/day → need sub-hourly partitions
With Iceberg, you change partitioning on the fly. Without it, you rewrite the entire table — potentially petabytes of data.
Hidden Partitioning
Iceberg's hidden partitioning means queries don't need to reference partition columns:
-- This query automatically prunes partitions
SELECT * FROM events WHERE event_time > '2026-03-01';
-- Iceberg knows to skip partitions outside this range
No WHERE partition_date = ... required. The query planner handles it.
Frequently Asked Questions
Can Delta Lake or Hudi do partition evolution?
No. Delta Lake and Hudi require rewriting all existing data to change partitioning. This makes partition evolution prohibitively expensive for large tables. Partition evolution is unique to Iceberg.
Does partition evolution affect query performance?
Not negatively. Iceberg's query planning handles mixed partitioning transparently. Each manifest file records which partitioning scheme was used for its data files.
