Streaming threat detection is the practice of evaluating security event streams continuously in real time, the moment events arrive, rather than after they are batch-indexed. Instead of waiting for logs to land in a SIEM before running queries, rules execute as SQL over live data, cutting detection latency from minutes to seconds.
How does streaming threat detection work?
Traditional security monitoring writes logs to a central store, waits for indexing to complete, then runs scheduled queries or correlation rules against the indexed data. That pipeline introduces inherent latency: indexing alone can take one to several minutes, and rules typically run on a schedule rather than continuously.
Streaming threat detection inverts this model. Security events (authentication logs, network flows, process telemetry, API call records) flow into a message broker such as Apache Kafka as they are generated. A streaming processing layer consumes that stream and evaluates detection rules continuously, computing aggregates, detecting patterns, and triggering alerts without waiting for any batch step.
The result is that the gap between an event occurring and a rule firing shrinks from minutes to seconds. For attacks that move quickly, such as credential stuffing or lateral movement, that difference in latency directly affects whether a response can contain the threat before damage is done.
What is the difference between streaming threat detection and a SIEM?
SIEMs such as Splunk and Microsoft Sentinel are built around a batch-index architecture. Events are ingested, written to a storage layer (Elasticsearch or a columnar store), indexed, and then made queryable. Detection rules written in SPL or KQL execute against this indexed store on a scheduled basis, often every one to five minutes.
The key differences are:
Detection latency. A SIEM rule fires at its next scheduled run, after indexing completes. A streaming rule fires as soon as the pattern is observed in the live event stream. For a brute-force attack generating hundreds of failed logins over 30 seconds, a streaming rule can alert within that window.
Rule language. SIEMs use proprietary query languages (SPL, KQL, SIGMA). Streaming threat detection with SQL uses standard ANSI SQL, including window functions and joins, which security engineers can learn quickly and reuse across tools.
Infrastructure cost. SIEMs charge by ingested volume and require substantial storage for the indexed copy of all logs. A streaming approach can evaluate rules on the live stream without necessarily storing everything, keeping costs proportional to the detection rules that matter rather than total log volume.
Historical correlation. SIEMs excel at retrospective investigation across months of history. Streaming is optimized for real-time detection. In practice, most production security pipelines use both: streaming for real-time alerting and a data lake for forensic investigation.
What security threats can streaming SQL detect?
Streaming SQL can detect a wide range of attack patterns because it can count, aggregate, and join events across a sliding or tumbling time window.
Brute force and credential stuffing. Count failed authentication events per source IP or username within a one-minute window. If the count exceeds a threshold, generate an alert. Streaming window functions make this a single SQL rule.
Lateral movement. Join authentication success events against an asset inventory table to identify accounts authenticating to an unusual number of hosts in a short period. Stream-table joins in SQL make this straightforward without writing application code.
Privilege escalation. Detect sequences where a process spawns a shell or a user transitions from a low-privilege role to an administrative one. Pattern matching over ordered event streams can surface these sequences in real time.
Data exfiltration. Aggregate outbound network flow volume per destination IP per hour. A sudden spike above baseline, joined against a threat intelligence feed of known bad IPs, produces a high-fidelity alert with minimal false positives.
Anomalous API access. Count API calls per user or service account per minute. Deviations from a rolling average, computed as a materialized view, flag compromised credentials or rogue automation.
Each of these patterns maps naturally to SQL aggregations, window functions, or joins, which means analysts can write and maintain rules without specialized CEP (complex event processing) expertise.
What does a streaming threat detection pipeline look like?
A typical pipeline has three layers.
Sources. Security events arrive from endpoints, cloud APIs, network sensors, and authentication systems. These events are published to Apache Kafka topics, one per event type (auth events, network flows, process events). Kafka provides durable buffering and fan-out to multiple consumers.
Processing layer. A streaming database subscribes to those Kafka topics and maintains SQL materialized views over the live stream. Each detection rule is a materialized view. When the rule condition is satisfied, the result appears in the view immediately. Window functions handle time-bounded aggregations; stream-table joins correlate live events against reference data such as asset inventories and threat feeds.
Alert output. Materialized view results are written to downstream systems: a SOAR platform for automated response, a ticketing system, or back into Kafka for further enrichment. Alert metadata can also be written to PostgreSQL-compatible endpoints, making integration with existing tooling straightforward.
This architecture keeps the detection logic in SQL, separate from the infrastructure plumbing. Adding a new detection rule means writing a new materialized view, not deploying new application code.
How does RisingWave enable streaming threat detection?
RisingWave is a PostgreSQL-compatible streaming database built in Rust and open source under Apache 2.0. It ingests from Kafka, CDC streams, and Pulsar natively, and exposes results through a standard PostgreSQL interface.
For streaming threat detection, RisingWave provides several concrete capabilities.
SQL detection rules as materialized views. Each detection rule is a CREATE MATERIALIZED VIEW statement. RisingWave maintains the view incrementally, recomputing only the affected aggregates when new events arrive. A brute-force detection rule looks like this:
CREATE MATERIALIZED VIEW brute_force_alerts AS
SELECT
source_ip,
username,
COUNT(*) AS failed_attempts,
window_start,
window_end
FROM TUMBLE(auth_events, event_time, INTERVAL '1' MINUTE)
WHERE status = 'FAILED'
GROUP BY source_ip, username, window_start, window_end
HAVING COUNT(*) > 10;
When the count of failed logins for a given source IP and username exceeds ten within any one-minute window, a row appears in brute_force_alerts. Downstream systems query this view via PostgreSQL wire protocol or consume it as a Kafka sink.
Exactly-once semantics. RisingWave guarantees exactly-once processing with state stored durably in S3. Alerts are not duplicated or missed during restarts or scaling events, which matters for security workflows where duplicate alerts create analyst fatigue and missed alerts create risk.
No Java, no separate cluster. Unlike Apache Flink, which requires Java or Scala, a separate cluster deployment, and RocksDB for hot state, RisingWave runs as a single deployment with familiar SQL. Security engineers do not need to maintain a JVM-based streaming cluster to operate detection rules.
PostgreSQL output to SIEM and SOAR. Alert tables are queryable over the PostgreSQL wire protocol. Existing SIEM connectors, SOAR platforms, and BI tools that speak PostgreSQL can read RisingWave results directly without custom integration code.
Frequently asked questions
Can RisingWave replace a SIEM entirely? Not typically. RisingWave is optimized for real-time streaming detection, not for long-term log storage and forensic investigation. Most teams use RisingWave alongside a SIEM: RisingWave for sub-minute alerting on live event streams, and the SIEM for historical search and compliance reporting.
How does RisingWave handle out-of-order events? RisingWave uses watermarks to handle late-arriving events. You configure an acceptable lateness window, and RisingWave holds aggregations open until the watermark advances past the window boundary. This is important for network flow data where events may arrive seconds or tens of seconds late.
Does streaming threat detection require custom code beyond SQL?
No. RisingWave surfaces a standard PostgreSQL interface. Detection rules are SQL materialized views. Integration with Kafka sources uses CREATE SOURCE statements. Engineers familiar with SQL can build and maintain a full detection pipeline without writing application code in Java, Python, or Scala.
Ready to run streaming threat detection with SQL? RisingWave is open source and free to use. Start with the RisingWave documentation or try RisingWave Cloud for a managed deployment with no infrastructure to operate.
