Background: Why Traditional Data Architectures Are Slowing Down Enterprise Transformation
Hivemind Technologies, a Germany-based tech company, specializes in helping industries like energy, manufacturing, and infrastructure achieve intelligent, data-driven transformations. Through their work building data platforms with clients, they identified a growing, critical issue: while the volume of data enterprises possess is constantly growing, the proportion actually being utilized is steadily declining.
From IoT devices in factories and sensors in urban infrastructure to user operation logs and energy consumption data, information is typically collected—and then siloed away in disparate systems. Many enterprises attempt to consolidate data flows by implementing so-called Medallion architectures: Bronze for raw data, Silver for cleaned and enriched data, and Gold for analytical outputs. However, these architectures often rely on offline batch processing, resulting in slow performance, complex structures, and difficult maintenance, often rendering them unsuitable for businesses with high real-time requirements.
This challenge presented a clear opportunity for Hivemind. They decided to break free from the traditional "scheduling + ETL + data landing" framework and build a new type of streaming, real-time, unified data infrastructure. This approach has been successfully implemented for several large clients, including Siemens.
Solution Selection: Why Hivemind Chose RisingWave for a Streaming Medallion Architecture
During the selection phase, Hivemind evaluated various real-time processing solutions, including Apache Flink, Kafka Streams, Spark Structured Streaming, and several mainstream ETL platforms. However, many of these tools proved too heavyweight, overly complex, or came with steep learning curves and high maintenance overhead.
Ultimately, they chose RisingWave—a distributed streaming database that supports standard PostgreSQL syntax. This system not only processes streaming data using SQL but also natively supports materialized views, stream aggregation, and multi-source input with multi-target output, making it a natural fit for the three layers of the Medallion architecture.
More importantly, the "real-time computation + real-time storage + multi-target delivery" combination offered by RisingWave significantly simplifies the complexity of deploying data platforms for enterprises. This enables Hivemind to deliver a modern data system to clients that is truly operational, queryable, and cost-effective, all with significantly reduced effort and time.
Architecture Design: Designing Bronze, Silver, and Gold Layers for Streaming Data
Hivemind has validated the feasibility of this design pattern across multiple projects. To illustrate how this architecture operates, let's consider an urban EV charging network as an example.
Bronze Layer: Ingesting Raw Streaming Data
Charging stations from various manufacturers send raw messages in real-time via Kafka. The data formats are inconsistent, fields vary, and even units can differ (e.g., temperature in Fahrenheit or Celsius). This raw data is directly ingested into RisingWave, forming a streaming Bronze layer that doesn't require data to be stored on disk or pre-processed.
Every data point can be processed downstream while retaining maximum fidelity, which facilitates traceability, auditing, and replay.
Silver Layer: Unifying, Cleaning, and Enriching Data in Real-Time with SQL
The Silver layer aims to transform this often disparate and raw data into standardized, structured, and usable information. The traditional approach involves periodically triggering Spark batch jobs for cleaning. In Hivemind's approach, RisingWave handles this entire step in real-time.
Instead of complex coding, engineers write SQL queries to:
Unify different field names (e.g.,
temp_c,temperature_f→temperature_c).Convert units to a consistent standard.
Enrich missing fields (e.g., converting latitude/longitude to addresses or adding weather information).
Filter out invalid data in real-time.
This allows engineers to focus on "business logic" rather than "scheduling logic." Cleaning rules are readable, maintainable, and can be hot-updated, significantly reducing complexity.
Gold Layer: Real-Time Aggregation, Insights, and Delivery—One Logic, Multiple Uses
The Gold layer focuses on aggregation and generating insights. In the EV charging scenario, examples include:
Total charging volume per charging station per hour.
Peak usage times.
Devices with abnormally high error rates.
Areas in a city with the heaviest power grid load.
RisingWave's materialized views generate all these metrics in real-time, eliminating the need for batch aggregation or intermediate caching layers. The results from these materialized views can be:
Directly supplied to visualization platforms for dashboards.
Synced to Iceberg tables for offline analysis.
Delivered to Kafka topics for downstream system subscription.
Stored in PostgreSQL or other OLAP systems for BI queries.
Architecturally, the Gold layer is a "result of streaming," not a "product of batching."
Application Outcomes: Successful Transformation for Siemens and Other Enterprises, Significant Cost Reduction
Hivemind has successfully deployed this streaming Medallion architecture for several large clients, with Siemens serving as a prominent example.
In the Siemens project, data was sourced from thousands of field devices and sensors. Their previous system relied on nightly batch jobs for data synchronization and cleaning, which resulted in long processing pipelines, high latency, and escalating costs. Migrating the entire logic to RisingWave yielded immediate improvements:
Data latency dropped from hours to seconds.
Cleaning logic transformed from complex script stacks to SQL rules, drastically reducing maintenance costs.
Infrastructure resource savings exceeded 50%, eliminating the need for dedicated scheduling clusters and intermediate data landing layers.
Data availability improved, allowing business departments to make real-time decisions based on views directly.
Crucially, this architecture also offers high flexibility and portability. It can run in the cloud or be deployed in on-premises data centers, fully aligning with enterprise demands for security, compliance, and operational control.
Summary: A Shift in Thinking, Not Just Technology
Hivemind's core strength isn't in developing databases. Instead, they apply an engineering perspective to pinpoint critical bottlenecks within enterprise data architectures. By leveraging RisingWave, they've crafted a solution that is simpler, more real-time, and offers greater control.
This approach is more than just assembling new tools; it signifies a fundamental paradigm shift in data architecture: moving from offline to real-time, from complex scripts to SQL, from rigid scheduling to continuous streaming, and from siloed stacks to integrated solutions.
While RisingWave provides the powerful engine for this streaming Medallion architecture, Hivemind is the essential partner that brings this technology to life, delivering tangible results in real-world enterprise settings.
Ultimately, the future of data infrastructure won't be judged by the complexity of its components, but by its ability to build truly flexible, real-time, and intelligent systems efficiently—using fewer resources and less overhead. Hivemind is already paving the way in this new direction.
Ready to Modernize Your Data Architecture?
Hivemind's success with the streaming Medallion architecture, powered by RisingWave, showcases a clear path to real-time data processing, simplified operations, and significant cost savings for modern enterprises. Imagine achieving sub-second latency and slashing infrastructure costs by over 50%, just like Siemens.
Ready to explore how RisingWave can do the same for you?
Discover RisingWave: Learn more about how our unified data platform can be the engine for your real-time data solutions.
Explore the Docs: Dive into our documentation to see how easy it is to build streaming pipelines, materialized views, and integrate with systems like Kafka.
Try RisingWave Today:
Download the open-sourced version of RisingWave to deploy on your own infrastructure.
Get started quickly with RisingWave Cloud for a fully managed experience.
Talk to Our Experts: Have a complex use case or want to see a personalized demo? Contact us to discuss how RisingWave can address your specific challenges.
Join Our Community: Connect with fellow developers, ask questions, and share your experiences in our vibrant Slack community.
We're excited to help you build a faster, more efficient, and intelligent data future with RisingWave!
