Real-Time Log Analytics Without ELK: A Streaming SQL Approach

The ELK stack (Elasticsearch, Logstash, Kibana) is the traditional choice for log analytics, but it's operationally heavy and expensive at scale. A streaming database offers an alternative: ingest logs from Kafka, compute aggregations and alerts with SQL, and query via PostgreSQL — replacing three systems with one.

Log Analytics with RisingWave

-- Ingest structured logs from Kafka
CREATE SOURCE app_logs (service VARCHAR, level VARCHAR, message VARCHAR,
  trace_id VARCHAR, response_time_ms INT, ts TIMESTAMP WITH TIME ZONE)
WITH (connector='kafka', topic='app-logs', properties.bootstrap.server='kafka:9092')
FORMAT PLAIN ENCODE JSON;

-- Error rate by service
CREATE MATERIALIZED VIEW error_rates AS
SELECT service,
  COUNT(*) as total_5min,
  COUNT(*) FILTER (WHERE level='ERROR') as errors_5min,
  COUNT(*) FILTER (WHERE level='ERROR')::DECIMAL / NULLIF(COUNT(*), 0) as error_rate
FROM app_logs WHERE ts > NOW() - INTERVAL '5 minutes'
GROUP BY service;

-- Slow request tracking
CREATE MATERIALIZED VIEW slow_requests AS
SELECT service, COUNT(*) as slow_count,
  AVG(response_time_ms) as avg_response,
  MAX(response_time_ms) as max_response
FROM app_logs WHERE response_time_ms > 1000 AND ts > NOW() - INTERVAL '5 minutes'
GROUP BY service;

ELK vs Streaming SQL

Aspect	ELK Stack	RisingWave
Components	Elasticsearch + Logstash + Kibana	Single system
Full-text search	✅ Excellent	❌ Not the focus
Real-time aggregations	Good (but expensive)	✅ Excellent (SQL MVs)
Cost at scale	High (Elasticsearch memory)	Lower (S3 state)
Alerting	Via Watcher/ElastAlert	Via SQL views
Query language	KQL/Lucene	Standard SQL

When to Use Which

ELK: When you need full-text log search ("find all logs containing error X"). RisingWave: When you need real-time log aggregations, error rate monitoring, and alerting with SQL.

Frequently Asked Questions

Can RisingWave replace Elasticsearch for logs?

Not entirely. Elasticsearch excels at full-text search across log messages. RisingWave excels at real-time aggregations and alerting over structured log data. Many teams use both: RisingWave for real-time metrics and alerting, Elasticsearch for log search and investigation.

How do I handle high-volume logs?

RisingWave scales horizontally across multiple compute nodes. For very high-volume logs (>100K events/sec), pre-filter in Kafka or at ingestion to process only the logs that matter for real-time metrics.