Schema in Streaming Data

In the context of streaming data, a Schema defines the structure, data types, and names of fields for the events or messages within a data stream. It acts as a contract that describes the format of the data being produced by sources and consumed by processing applications or sinks.

Having a well-defined schema is crucial for ensuring data quality, interpretability, and enabling robust data processing pipelines.

Core Components of a Schema

A schema typically specifies:

Field Names: The identifier for each piece of data within an event (e.g., user_id, product_name, timestamp, price).
Data Types: The type of data each field holds (e.g., STRING, INTEGER, FLOAT, BOOLEAN, TIMESTAMP, ARRAY, STRUCT/RECORD).
Order of Fields: In some serialization formats, the order of fields can be significant.
Constraints/Properties (Optional):
- Whether a field is required or optional (nullable).
- Default values for optional fields.
- Documentation or descriptions for fields.
- Logical types (e.g., an INTEGER field representing a DATE).

Importance of Schemas in Streaming

Data Validation: Schemas allow producers and consumers to validate whether data conforms to the expected structure and types, preventing data corruption or processing errors.
Interoperability: They enable different systems or microservices (potentially written in different languages) to understand and process data from the same stream correctly.
Data Interpretation: Provide a clear understanding of what each field represents, which is essential for developers and analysts.
Code Generation: For some serialization formats (like Avro or Protobuf), schemas are used to generate code (data access objects or classes) that simplifies data serialization and deserialization.
Schema Evolution: A well-managed schema system allows for changes to the data structure over time (e.g., adding new fields, deprecating old ones) while maintaining compatibility between producers and consumers.

Schema Definition and Management

Explicit Definition: Schemas are often explicitly defined using a schema definition language (e.g., Avro's .avsc files, Protobuf's .proto files, JSON Schema).
Implicit Schema (Schema-on-Read): For formats like JSON or CSV without an external schema definition, the schema might be inferred at the time of reading ("schema-on-read"). This can be flexible but is more prone to errors if data structures are inconsistent.
Schema Registry: In many streaming ecosystems (especially those using Kafka with Avro or Protobuf), a Schema Registry is used as a centralized service to store, manage, version, and validate schemas.
- Producers register schemas with the registry before sending data.
- Consumers fetch schemas from the registry to deserialize incoming data.
- The registry can enforce compatibility rules for schema evolution.

Schema Evolution

Data streams are often long-lived, and the structure of the data (the schema) may need to change over time. Schema Evolution refers to managing these changes gracefully. Common types of schema changes include:

Adding a new field (often with a default value or as an optional field).
Removing an existing field (usually requires it to be optional first).
Renaming a field (can be complex).
Changing the data type of a field (can be risky and may require careful planning).

Schema registries often enforce compatibility rules (e.g., backward, forward, or full compatibility) to ensure that schema changes don't break existing producers or consumers.

Schemas in RisingWave

RisingWave, as a streaming database, heavily relies on schemas:

CREATE SOURCE: When defining a data source (e.g., from Kafka), you must specify the schema of the incoming data, including column names and data types. This allows RisingWave to parse and interpret the data correctly.
- For formats like Avro or Protobuf, RisingWave can integrate with a Schema Registry (like Confluent Schema Registry or a Kafka schema registry with Upsolver schema registry API compatibility) to fetch schemas, or you can provide the schema definition directly (e.g., via a message and schema location for Protobuf).
CREATE TABLE: Used to define tables within RisingWave, which also have a fixed schema.
CREATE MATERIALIZED VIEW / CREATE SINK: The schema of materialized views and sinks is derived from the SQL query defining them.
Data Type System: RisingWave has its own SQL data type system, and incoming data is mapped to these types.