Company Background
Company A is a hedge fund with a scale of ten billion dollars, specializing in quantitative investment. With an employee count approaching 100, Company A’s investment research team comprises a significant 70%. The team has built forecasting models by leveraging modern machine learning technologies and has developed a low-latency trading system for fully automated algorithmic trading. These core competencies have enabled rapid development, with the company now managing over 10 billion dollars in assets.
Before using RisingWave
As Company A’s business expands rapidly, the array of managed fund products has seen a steady rise, paralleled by the escalating volume of account transaction data that needs to be processed daily. In the field of quantitative trading, low latency stands as a pivotal factor. Real-time and rapid analysis of market conditions and transaction flows can facilitate quicker and more effective decision-making, thereby enhancing the company’s ability to capture trading opportunities.
As early as 2021, Company A used ksqlDB for real-time alert functions but soon found that, due to the limitations in its design, it could not be applied across various scenarios. So later, the company transitioned to the well-established stream processing engine, Flink, to connect different data systems for ETL real-time calculations. Flink's DataStream API describes stream computing in an imperative language, whose flexibility and rich components can implement more complex data processing logic. However, it also brings higher learning, development, and operational costs for developers unfamiliar with the JVM stack.
Therefore, while Flink aligns with its requirements, Company A continued its quest for a stream computing system better tailored to its business needs. Its primary considerations revolve around the challenges presented in the following aspects:
Learning cost
Company A has distinct roles such as researchers, traders, and system developers, most of whom do not have a JVM programming background. Therefore, in terms of ease of use, the learning cost of the stream computing system is very important, making it difficult for Flink to be widely promoted in the company's business lines.
Resource cost
In a live trading colocation data center (which commonly means servers hosted near the exchange in quant trading), hardware resources are scarce. Thus, factors such as lower resource occupancy and lower deployment complexity are crucial considerations for Company A.
Observability
Metrics monitoring systems and state query capabilities are very powerful assistants in latency tuning and system debugging. A minor logic change can cause significant changes in CPU and memory usage. Therefore, a comprehensive visualization monitoring system and state query capabilities are especially crucial for Company A.
Technical support
For beginners in stream computing, there are many technical and conceptual confusions, especially during the PoC (Proof of Concept) project phase. Thus, efficient communication with the technical team and receiving high-quality response are also important for Company A, as both of them quickly deepen the company’s understanding of the system.
Why choose RisingWave
Through its experiences with ksqlDB and Flink, Company A has experienced their respective features and advantages. However, the company were still searching for a more outstanding system that suited its business scenarios until it encountered RisingWave. Through in-depth usage, Company A found that RisingWave is an exceptionally modern stream computing system. As highlighted on its official website, in a sense, it indeed redefines stream computation. The following features of RisingWave particularly captured Company A’s attention:
PostgreSQL-compatible
What left a lasting impression on Company A was RisingWave's strategic choice to employ PostgreSQL for articulating stream computations from the beginning. SQL, as a declarative language, is arguably one of the most widely used forms of expression. Its simplicity, ease of comprehension, and high-level abstract syntax allow developers to focus more on the essence of business logic, significantly reducing learning and development costs, and thereby improving overall development efficiency.
Because RisingWave is compatible with the Postgres protocol, the technology stack within the Postgres ecosystem seamlessly integrates with RisingWave, incorporating numerous capabilities seamlessly. From Company A’s practice, SQL's expressive power can already cover the vast majority of business scenarios. For exceedingly complex and customized data processing, RisingWave offers the flexibility of User-defined Functions (UDFs) or other imperative languages.
Low latency
As mentioned earlier, low latency is a very important consideration for technology selection in the field of quantitative trading, under the premise of meeting the business functions.
Leaving aside extremely demanding scenarios for ultra-low latency, Company A uses RisingWave in other scenarios like real-time expansion of transaction flows with dimension tables, as well as real-time position accumulation, achieving an end-to-end latency of 5-10ms level. Even more complex real-time queries, such as Profit & Loss and other monitoring metrics calculations, can also be completed within a few hundred of milliseconds.
Data integration
RisingWave offers a rich set of sources and sink connectors, ranging from mainstream message queues to OLTP and OLAP databases, and to data lake storage engines. It also offers mainstream encoding methods like JSON, Avro, and Protobuf, facilitating the convenient and cost-effective integration of existing business data systems into RisingWave. If Kafka is acknowledged as the biggest winner in the stream processing market, then RisingWave can also be described as the Swiss Army knife in the field of stream computing.
Abundant operators
In Company A’s projects deployed to production environment, more than 50 tables, materialized views, and source/sink connectors interacting with external systems have been created. Also, operators such as temporal joins, interval joins, temporal filters, TopN operators, materialized views that use real-time incremental calculation to improve freshness, time window functions, watermark mechanism that supports out-of-order processing are all unique to RisingWave. It also supports most of the syntax in PostgreSQL, such as common table expression, DISTINCT ON, etc.
With such abundant operators, RisingWave can meet Company A’s needs in scenarios, such as the expansion of fact tables with dimension table, multi-stream aggregation, real-time ETL, window metrics statistics, and so on.
Observability
RisingWave brings very friendly and comprehensive observability to streaming jobs. First, it provides a range of metrics of different granularities through Grafana dashboards, making it well-suited for tuning and metrics monitoring in production. Second, the internal state of streaming jobs (like a table) can be queried with SQL statements, which is very helpful for debugging. Additionally, its compatibility with the Postgres protocol also enables RisingWave’s direct connection to tools in the Postgres ecosystem, greatly simplifying SQL debugging and troubleshooting of data-related issues.
Technical support
The RisingWave team is professional, enthusiastic, and patient, providing Company A with highly efficient, quick, and meticulous assistance in the early stages of its use, which enabled it to gain a deeper understanding of RisingWave and stream computing. With the RisingWave team’s support, Company A has successfully implemented some business use cases during the POC stage.
Implementation of RisingWave
Company A’s collaboration with RisingWave started a few months ago. At that time, Company A encountered numerous trading scenarios where highly efficient low-latency table expansion, real-time aggregation of multiple data streams, real-time metrics calculations, and interactions with various data systems (like message queues and databases) were required.
The company’s initial use of Flink did not achieve the expected effect. So it looked for new solutions and, considering the costs of development, operation, and maintenance, hoped to avoid using other imperative languages for development in its new solution. That’s why it chose RisingWave.
Under the careful guidance of the RisingWave team, Company A managed to complete the development of the following businesses using SQL declarative language and apply it to the production environment:
- Ingesting real-time transaction flows from message queues, expanding them based on dimension tables, and sinking the data to downstream data systems
- Accumulating account positions in real time based on transactions, and sinking the data to downstream data monitoring systems
- Calculating account Profit & Loss in real time based on transactions and positions, and sinking the data to downstream monitoring systems
- Aggregating multiple event streams, including market prices and transaction flows, into derived messages, and sinking the data to downstream data systems
>
After comprehensive learning and hands-on experience with RisingWave, Company A has seamlessly integrated it into the company’s live trading production environment, where RisingWave has demonstrated consistent stability. > >
>
Emerging as a performant solution, RisingWave redefines stream processing with a unique perspective, centering around its core idea of compatibility with PostgreSQL. This feature facilitates the seamless integration of diverse data systems, simplifying the complexity of data streams. Beyond its rich functional components, RisingWave stands out for its exceptional advantages in terms of ease of use, low latency, minimal learning costs, and streamlined deployment and operational costs. Furthermore, it continues to evolve and improve. > >
>
RisingWave has effectively addressed the challenges Company A previously faced, pushing the boundaries of stream processing. It empowers Company A to confidently navigate through more innovative, complex, and demanding business scenarios. > >
