API World + CloudX + Dev Innovation Summit 2024

Rayees Pasha, RisingWave Labs, Head of Product

In this talk, I focus on addressing the high latency issues associated with Cloud storage in data processing systems that employ a decoupled compute and storage architecture. We delve into the root causes of latency in this context and explore various techniques to minimize the impact of object storage latency on stream processing performance. Our proposed approach is to implement a tiered storage mechanism that leverages a blend of high-performance and low-cost storage tiers to reduce data movement between the compute and storage layers while maintaining efficient processing.

We will present experimental results that demonstrate the effectiveness of our approach in mitigating the impact of Cloud storage latency on data processing. By the end of the talk, attendees will have gained insights into how to optimize their data processing systems for reduced latency and improved cost-efficiency.