Supporting streaming deployments with Qwak

Qwak now supports deploying machine learning (ML) models with event-driven streaming architecture using Apache Kafka to support high-throughput predictions.

This new capability allows data scientists to deploy with a click ML models as an endpoint to receive data as stream and output predictions as streams.

Real-time ML inference at scale has become an essential part of modern applications. Although we started our deployment service with support of real-time predictions based on a web server, we do see among our customers a high demand for streaming-based ML predictions.

Streaming inference is useful in the following cases:

• When the inference requests should triggered by an already existing stream of messages
• When you would like to decouple the caller from the model
• When you need to handle with prediction service failures because of high prediction traffic

How it works

Once a model is deployed to Qwak using the Streaming option, the deployed model will be triggered when the producer topic receives features/inference requests, and then it pushes the prediction to a consumer topic.

Once the model is deployed, you can track the service health metrics and be alerted if the metrics are above/below certain thresholds, such as error percentage, average throughput, consumed messages, consumer lag, processing lag, and errors over time.

Getting started with Qwak Streaming deployment:

Using Qwak Management console:

Choose the number of pods and CPU/memory size of the pod, and then add the address of the Bootstrap server and consumer/producer topic names.

Qwak CLI command: 

qwak models deploy stream \

 --model-id "demo_stream" \

 --build-id "my_build_id" \

 --consumer-bootstrap-server "kafka-bootstrap-server.svc.cluster.local" \

 --consumer-topic "in-topic" \

 --consumer-group "consumer-group-example" \

 --consumer-auto-offset-reset latest \

 --consumer-timeout 60000 \

 --producer-bootstrap-server "kafka-bootstrap-server.svc.cluster.local" \

 --producer-topic "out-topic" \

 --producer-compression-type gzip \

 --workers 2

Qwak streaming deployment is perfect for event-based predictions that require high throughput, low latency, and fault tolerant environments. 

Get started for free today!