Serialization Format

A Serialization Format (often called a "wire format") defines the rules and conventions for converting structured data (like objects, records, or messages) into a sequence of bytes for storage or transmission across a network, and for converting those bytes back into a usable structured format (deserialization).

In the context of data streaming and distributed systems, the choice of serialization format is crucial as it impacts performance, data size, schema evolution capabilities, and interoperability between different services and programming languages.

Key Considerations When Choosing a Format

• Performance (Speed): How quickly can data be serialized and deserialized? CPU-intensive formats can become bottlenecks.
• Data Size (Compactness): How much space does the serialized data occupy? Smaller sizes reduce storage costs and network bandwidth usage. Binary formats are generally more compact than text-based ones.
• Schema Evolution: How well does the format support changes to the data structure (schema) over time without breaking existing producers or consumers?
• Human Readability: Are the serialized bytes human-readable (like JSON or XML) or binary (like Avro or Protobuf)? Readability can aid debugging but often comes at the cost of size and performance.
• Language Interoperability: Does the format have good support and libraries across different programming languages?
• Ease of Use: How complex is it to define schemas and work with the serialization libraries?

Common Serialization Formats in Streaming

1. JSON (JavaScript Object Notation):

   • Pros: Human-readable, widely supported, flexible (often schema-less, though JSON Schema can be used).
   • Cons: Verbose (larger data size), relatively slow to parse, schema evolution needs careful management (often ad-hoc or via JSON Schema).
   • Use Cases: APIs, configuration files, simple message passing where readability is key.

2. Apache Avro:

   • Pros: Compact binary format, rich data structures, robust schema evolution capabilities (schemas are typically stored or referenced externally, e.g., in a Schema Registry), good language support, particularly strong in the Hadoop/Kafka ecosystem.
   • Cons: Not human-readable, requires schema for deserialization.
   • Use Cases: High-volume event streaming with Kafka, data archiving, inter-service communication where schema management is important. RisingWave has strong support for Avro.

3. Protobuf (Protocol Buffers):

   • Pros: Very compact and efficient binary format (developed by Google), strong schema definition language (.proto files), generates optimized code for serialization/deserialization in multiple languages, good schema evolution support.
   • Cons: Not human-readable, requires schema for deserialization and code generation step.
   • Use Cases: High-performance RPC, microservices communication, event streaming where performance and compactness are critical. RisingWave supports Protobuf.

4. CSV (Comma-Separated Values):

   • Pros: Simple, human-readable (for tabular data), widely supported by spreadsheets and data tools.
   • Cons: No built-in schema support, limited data type representation, parsing can be ambiguous (e.g., escaping commas), not ideal for complex nested structures.
   • Use Cases: Simple tabular data exchange, bulk data loading.

5. Plain Text / Bytes:

   • Pros: Simple for unstructured or custom binary data.
   • Cons: No inherent structure or schema, requires custom parsing logic, poor interoperability unless a strict convention is followed.
   • Use Cases: Logging (though structured logging is often preferred), raw sensor data, specialized binary protocols.

RisingWave and Serialization Formats

When defining SOURCEs (for data ingestion) or SINKs (for data egress) in RisingWave, you specify the serialization format of the data. RisingWave's connectors use this information to correctly interpret incoming byte streams or to format outgoing data. Common formats supported by RisingWave for sources like Kafka include:

• JSON
• Avro (often in conjunction with a Schema Registry)
• Protobuf (often in conjunction with a Schema Registry)
• Debezium (which itself uses JSON or Avro for its message payload)
• Maxwell (CDC format)
• Canal (CDC format)

The choice of format for your data streams feeding into RisingWave will depend on your specific upstream systems, performance requirements, and schema management strategy.