Event Time
Event Time is the timestamp representing when an event actually occurred at its source system or device, before it even entered the stream processing pipeline. It's embedded within the event's data payload itself (e.g., a sensor reading timestamp, a transaction timestamp, a log entry timestamp).
Event Time processing allows for analyzing events based on when they actually happened, regardless of delays in ingestion or processing. This is crucial for achieving deterministic and accurate results, especially when dealing with Windowing over time or joining streams based on when events occurred relative to each other.
Event Time vs. Other Timestamps
It's important to distinguish Event Time from:
- Processing Time: The time when an event is processed by a specific operator or task within the stream processing system. This is based on the local clock of the processing machine. Processing time is simpler to implement but sensitive to processing delays, system load, and failures, leading to non-deterministic results across different runs or environments.
Example:
- A sensor logs a temperature reading at 10:00:00 AM (Event Time).
- The stream processor reads the event from Kafka and processes it in a window operator at 10:00:10 AM (Processing Time).
For accurate windowed analysis (e.g., calculating the average temperature between 10:00 and 10:05), using Event Time (10:00:00 AM) is essential. Processing Time (10:00:10 AM) would place the event in the wrong window.
Importance and Challenges
- Deterministic Results: Processing based on Event Time ensures that the results of analyses (especially time-windowed ones) are consistent and reproducible, regardless of when the processing actually occurs or how much latency is present.
- Handling Out-of-Order Events: Due to network delays, distributed sources, or retries, events often arrive at the processor out of their original event time order. Stream processors need mechanisms to handle this disorder, typically by buffering events and using Watermarks.
- Defining Completeness: How long should the system wait for potentially late-arriving events for a given time window? This is where Watermarks come in, providing a notion of time progress and allowing the system to close windows and emit results.
Event Time in RisingWave
RisingWave fully supports Event Time processing:
- Event Time Extraction: When creating a Source or table, users typically designate a timestamp column within the incoming data as the Event Time attribute. RisingWave extracts this timestamp from each record.
- Watermarks: RisingWave uses Watermarks to track the progress of Event Time and handle out-of-order events. Watermarks are generated based on the observed event times, providing a guarantee that (ideally) no events with an event time earlier than the watermark will arrive later. This allows RisingWave to trigger computations, close windows, and emit results based on Event Time completeness. Users define watermark strategies (e.g., WATERMARK FOR event_time_column AS event_time_column - INTERVAL '5' SECOND) when creating sources or tables.
- Windowing: RisingWave's time Windowing functions (Tumbling, Hopping, Sliding) operate primarily on Event Time when watermarks are defined, ensuring accurate, deterministic window assignments based on when events actually happened.
By leveraging Event Time and Watermarks, RisingWave enables accurate stateful stream processing based on the actual occurrence time of events.
Related Glossary Terms
