Protobuf

Protobuf, short for Protocol Buffers, is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. Developed by Google, it's akin to JSON or XML but is typically smaller, faster, and simpler. You define how you want your data to be structured once using a special definition language in .proto files, and then you can use generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Key Concepts

.proto Files: These files define the structure of your data, called "messages." A message is a collection of named fields, where each field has a type (e.g., string, integer, boolean, another message type).
```
syntax = "proto3";

message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 results_per_page = 3;
}
```
Message Compiler (protoc): The Protobuf compiler (protoc) takes your .proto files and generates source code in your chosen language (e.g., Java, Python, C++, Go, C#, Ruby, JavaScript). This generated code includes classes for your messages and methods to serialize (write) and deserialize (read) them.
Binary Format: Protobuf messages are serialized into a compact binary format. This makes them efficient for network transmission and storage compared to text-based formats like JSON or XML.
Schema Evolution: Protobuf is designed to allow for backward and forward compatibility as your data structures evolve. You can add new fields or deprecate old ones without breaking existing producers or consumers, provided you follow certain rules (e.g., new fields should be optional, field numbers must not be reused).

How it Works

Define Schema: You create .proto files defining your message types.
Compile: You use the protoc compiler to generate data access classes in your desired programming language(s).
Serialize: In your application, you create instances of the generated message classes, populate their fields, and then call a serialization method (e.g., serializeToString(), toByteArray()) to convert the message object into a binary byte array or string.
Transmit/Store: This binary data is sent over the network or stored in a file or database.
Deserialize: The receiving application (or the application reading from storage) uses the same generated message class to parse the binary data back into a structured message object (e.g., using a parseFromString() or parseFromByteArray() method).

Advantages

Efficiency: Compact binary format leads to smaller message sizes and faster serialization/deserialization compared to JSON or XML.
Strong Typing & Schema Enforcement: Schemas are explicitly defined, reducing ambiguity and errors.
Language Interoperability: Supports code generation for many popular programming languages.
Schema Evolution: Well-defined rules for evolving schemas while maintaining compatibility.
Code Generation: Automates the creation of data access code, reducing boilerplate.

Disadvantages

Not Human-Readable: The binary format is not directly human-readable like JSON or XML, which can make debugging harder without tools.
Requires Compilation Step: Changes to .proto files require recompilation and regeneration of code.
External Dependency: Requires the Protobuf compiler and runtime libraries.

Use in Streaming Systems (like RisingWave)

Protobuf is a popular choice for data serialization in streaming systems, especially when used with platforms like Apache Kafka:

Data Ingestion: RisingWave can ingest data from sources (e.g., Kafka topics) where messages are serialized using Protobuf.
- To do this, you typically provide the schema of the Protobuf messages. This can be done by referencing a compiled .proto file (as a .pb or descriptor set file) or by connecting to a Schema Registry (like Confluent Schema Registry) that manages Protobuf schemas.
Data Egress (Sinks): While less common for RisingWave's primary use cases (which often involve serving materialized views via SQL or direct changefeeds), it's conceivable for a sink to serialize data into Protobuf format for other systems.

Protobuf vs. Avro vs. JSON

Protobuf vs. Avro: Both are binary, schema-based formats. Avro schemas are typically defined in JSON and are often packaged with the data or retrieved from a schema registry. Protobuf relies on pre-compiled .proto files. Both offer good schema evolution. Avro is often favored in Hadoop ecosystems, while Protobuf is prevalent in gRPC and other Google-originated technologies.
Protobuf vs. JSON: JSON is text-based, human-readable, and schema-less (though JSON Schema exists). Protobuf is binary, not human-readable without tools, and strictly schema-based. Protobuf is generally more efficient in terms of size and speed. JSON is often simpler for ad-hoc use and debugging.

RisingWave supports Protobuf (along with other formats like Avro and JSON) for data ingestion, allowing users to work with a variety of existing data sources and serialization standards.