Kafka Partitioning Strategies: How to Choose the Right Partition Key
Kafka exactly-once delivery guarantees each message is produced and consumed exactly one time, even during failures. This is achieved through idempotent producers, transactional APIs, and consumer offset management.
Three Levels
| Level | Mechanism | Scope |
| Idempotent producer | Producer deduplication (PID + sequence number) | Single partition |
| Transactional producer | Atomic multi-partition writes | Across partitions and topics |
| End-to-end exactly-once | Transactions + consumer offset commit | Full pipeline (read-process-write) |
How It Works
1. Producer assigns sequence number to each message
2. Broker deduplicates based on (ProducerID, PartitionID, SequenceNumber)
3. Transactional producer wraps multiple writes in a transaction
4. Consumer commits offsets within the same transaction
5. Result: exactly-once from source to sink
Configuration
# Producer
enable.idempotence=true
transactional.id=my-app-txn-1
# Consumer (Kafka Streams)
processing.guarantee=exactly_once_v2
Performance Impact
Exactly-once adds 5-50ms latency and reduces throughput by 10-30% due to transaction coordination. For most workloads, this trade-off is worth the correctness guarantee.
Frequently Asked Questions
Is Kafka exactly-once reliable in production?
Yes. Kafka's exactly-once implementation has been production-stable since Kafka 2.5 (2020). Major companies use it for financial transactions and billing systems.
Does RisingWave support Kafka exactly-once?
RisingWave provides exactly-once consumption from Kafka through its own checkpoint-based mechanism, independent of Kafka transactions.
