How Dustess Unlocks Real-Time Data Enrichment by Harnessing RisingWave

Company Overview

In business operations, Customer Relationship Management (CRM) plays a crucial role. Dustess is a leading SCRM (Social Customer Relationship Management) provider.

They offer a one-stop customer operation management platform that combines traditional CRM with social platform friend relationships to provide more comprehensive customer insights, more accurate decision analysis, and more effective customer operation tools.

Their mission is to help businesses attract customers, convert sales, and maintain customer relationships, thereby driving continuous business growth.

Challenges

Before adopting RisingWave, Dustess SCRM faced some significant challenges. Data enrichment is a common business processing scenario in data integration. However, when table data volumes reach tens of millions to billions, and the number of dimension tables exceeds a dozen, it presents a series of severe tests for the performance, cost, and stability of the entire data pipeline.

The Dustess team, with their extensive experience, used various optimization techniques, such as manually rewriting queries and performing incremental processing using batch processing engines. However, under this solution, data freshness was still limited, at approximately 1 hour. This clearly did not meet their need for real-time data. While batch processing is effective in some cases, it is not the best choice for real-time data . Stream processing can better achieve frequent updates to data state, ensuring rapid results refreshment.

Solution

Reasons for Choosing RisingWave

As experienced Flink users, the Dustess team had a deep understanding of their stream processing requirements. Initially, they became highly interested in the internal benchmark results of RisingWave compared to Apache Flink. They were searching for a stream system that could maintain JOIN states of hundreds of gigabytes without sacrificing overall efficiency. For a long time, they hadn't found a solution that met their needs, and they had to sacrifice data freshness by using batch processing.

Implementation Process

The initial collaboration began in June, and the Dustess team and the RisingWave team conducted a three-month PoC (Proof of Concept) evaluation. During this period, they gained confidence in the cost-effectiveness and feature completeness of RisingWave. Since the Dustess business's primary focus was on performance, the RisingWave team provided many performance optimization suggestions and helped identify and resolve bottleneck issues.

The Dustess team used dbt in their data warehouse workflow, and RisingWave provided native dbt support, making it easy for them to migrate data streams initially used for data warehousing.

Results

After adopting RisingWave, Dustess SCRM achieved impressive results: data freshness significantly reduced to less than 10 seconds, meeting their real-time data requirements.

In addition to strong support for streaming joins in data enrichment scenarios, RisingWave also met Dustess SCRM's various needs.

RisingWave greatly improved development efficiency, largely thanks to RisingWave's compatibility with PostgreSQL for materialized view syntax, which exceeded their expectations. Additionally, with the help of the RisingWave native dbt driver, rapid development was possible, and the cloud-native architecture significantly reduced maintenance and hardware costs. Advanced syntax such as window functions for complex calculations, handling JSONB types for semi-structured data... We'll cover these aspects in the future.