Streaming Aggregation - The Spark Behind Real Time ML

Efficient handling of multiple small and long time windows.
Gal Lushi
Gal Lushi
Tech Lead, Data Platform at Qwak
at
at
at

Real time ML tasks, such as credit card fraud detection, recommendation systems, anomaly detection and others use features that are computed as aggregates over real-time data streams.

Data freshness and low latency are key in these use cases, yet achieving them often results in a resource-intensive solution and an angry CFO.

Join Gal Lushi and Yoni Ben-Dayan as they take us through how Qwak’s engineering teams built a turn-key streaming aggregation solution that addresses challenges, while guaranteeing:

  • High data freshness, low latency and high throughput
  • EXACTLY ONCE and support for late arrivals.
  • Efficient handling of multiple small and long time windows.
  • Consistency between inference and training (because who wants a training-serving skew?)

Real time ML tasks, such as credit card fraud detection, recommendation systems, anomaly detection and others use features that are computed as aggregates over real-time data streams.

Data freshness and low latency are key in these use cases, yet achieving them often results in a resource-intensive solution and an angry CFO.

Join Gal Lushi and Yoni Ben-Dayan as they take us through how Qwak’s engineering teams built a turn-key streaming aggregation solution that addresses challenges, while guaranteeing:

  • High data freshness, low latency and high throughput
  • EXACTLY ONCE and support for late arrivals.
  • Efficient handling of multiple small and long time windows.
  • Consistency between inference and training (because who wants a training-serving skew?)

Qwak optimizes AI in production

“From our very first interaction, it was clear that Qwak understood our needs and requirements. Their platform enabled us to deploy a complex recommendations solution within a remarkably short timeframe. Moreover, Qwak is an exceptionally responsive partner, continually refining their solution.”
Lightricks
“We ditched our in-house platform for Qwak. I wish we had found them sooner.”
Upside
“Qwak streamlines AI development from prototype to production, freeing us from infrastructure concerns and maximizing our focus on business value.”
Notion