Schema Registry

A Schema Registry is a centralized service for managing and validating schemas, especially for data serialization formats like Apache Avro, Protobuf (Protocol Buffers), and JSON Schema, which are commonly used in event streaming platforms like Apache Kafka. It acts as a store for schemas and provides a way for producers and consumers of data to share and evolve schemas in a controlled manner.

Core Functions and Importance

• Centralized Schema Storage: Stores all versions of schemas for different data topics or subjects. Each schema is typically identified by a subject name (often related to a Kafka topic) and has multiple versions.
• Schema Validation: Enforces rules about how schemas can evolve. When a producer tries to register a new schema version, the registry checks it against configured compatibility rules (e.g., backward, forward, full compatibility) with previous versions. This prevents breaking changes that could disrupt consumers.
• Schema Distribution: Consumers can retrieve schemas from the registry by ID or by subject and version to correctly deserialize incoming messages. Producers might send a schema ID with the message, allowing consumers to fetch the exact schema used for serialization.
• Decoupling Producers and Consumers: Producers can evolve schemas independently of consumers (within compatibility rules), and consumers can adapt to new schema versions at their own pace.
• Data Governance: Provides a clear record of schema versions and their evolution, aiding in data governance and understanding data lineage.
• Efficiency: Serializing with schema IDs instead of embedding full schemas in every message can reduce message size, especially for binary formats like Avro and Protobuf.

How it Works (Typical Flow with Kafka & Avro)

1. Producer:

   • When a producer application wants to send a message to a Kafka topic, it first checks if the schema for that message is registered.
   • If not, or if it's a new version, the producer attempts to register the schema with the Schema Registry.
   • The registry validates the schema against compatibility rules. If valid, it stores the schema and assigns it a unique ID.
   • The producer then serializes the data using the schema and typically includes the schema ID in the message payload (or as metadata).
   • The message is sent to Kafka.

2. Consumer:

   • When a consumer application reads a message from Kafka, it extracts the schema ID.
   • It queries the Schema Registry using this ID to fetch the corresponding schema.
   • The consumer then uses this schema to deserialize the message payload.
   • If the schema is not cached locally, the consumer will fetch it from the registry; otherwise, it uses its cached version.

Common Schema Registries

• Confluent Schema Registry: A popular implementation often used with Apache Kafka.
• Apicurio Registry: An open-source registry supporting various schema types and storage backends.
• Cloud provider specific registries (e.g., AWS Glue Schema Registry).

RisingWave and Schema Registry

When RisingWave ingests data from sources like Kafka, especially when using formats like Avro or Protobuf, it can integrate with a Schema Registry.

• Source Definition: When creating a SOURCE in RisingWave, you can specify the Schema Registry URL and other relevant parameters.
• Deserialization: RisingWave's connectors will then communicate with the Schema Registry to fetch the appropriate schemas needed to deserialize the incoming messages from Kafka topics. This allows RisingWave to correctly interpret the structure and data types of the streaming data.

This integration simplifies data ingestion from schema-managed sources and ensures that RisingWave can adapt to schema evolution in the upstream systems.

Related Glossary Terms

• Schema in Streaming Data
• Schema Evolution in Data Lakehouses
• Avro
• Protobuf
• Apache Kafka
• Source
• Serialization Format
• Connector