RocksDB in Stream Processing: How State Backends Work
Apache Avro, Protocol Buffers (Protobuf), and JSON are the three main serialization formats for streaming data. Avro dominates in the Kafka ecosystem (with Schema Registry). Protobuf leads in gRPC and internal services. JSON is simplest but least efficient.
Comparison
| Feature | Avro | Protobuf | JSON |
| Encoding | Binary | Binary | Text |
| Schema | Embedded or registry | .proto files | None (self-describing) |
| Schema evolution | ✅ (add/remove fields) | ✅ (field numbers) | ❌ (no enforcement) |
| Size | Compact | Most compact | Largest |
| Speed | Fast | Fastest | Slowest |
| Human readable | ❌ | ❌ | ✅ |
| Schema Registry | ✅ (Confluent) | ✅ (Confluent) | ✅ (JSON Schema) |
| Streaming support | ✅ All engines | ✅ Most engines | ✅ All engines |
Recommendations
- Kafka pipelines: Avro with Schema Registry (industry standard)
- gRPC services: Protobuf (native format)
- Quick prototyping: JSON (human readable, no schema setup)
- Performance critical: Protobuf (smallest, fastest)
RisingWave Support
RisingWave supports all three: FORMAT PLAIN ENCODE JSON, FORMAT PLAIN ENCODE AVRO, FORMAT PLAIN ENCODE PROTOBUF.
Frequently Asked Questions
Which format should I use for Kafka?
Avro with Confluent Schema Registry is the industry standard for Kafka. It provides schema evolution, compact encoding, and schema enforcement. Use Protobuf if your organization is already standardized on it.
Is JSON too slow for streaming?
For most workloads, JSON performance is acceptable. At very high throughput (>100K events/sec), binary formats (Avro, Protobuf) provide measurable benefits in serialization speed and message size.
