# Real-Time Event Processing for AI/ML with Numaflow

Sri Harsha Yayi, Intuit | DE4AI 2024 | 22:38

Source: https://www.youtube.com/watch?v=-EAexlXbN1I
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/real-time-event-processing-for-ai-ml-with-numaflow
Published: 2024-10-09
Tags: data-pipelines, deployment, model-serving

## TL;DR
- Numaflow lets teams define streaming workloads as Kubernetes-native pipelines without managing connections to each event source and sink.
- Teams can write processing and inference logic in Python, Java, Go, or Rust while Numaflow handles data movement between pipeline components.
- Numaflow scales pipeline components using upstream backlog and processing rate, rather than relying only on CPU and memory usage.

## Summary
Sri Harsha Yayi introduces Numaflow, an open-source, Kubernetes-native platform from Intuit for real-time event processing and inference. He begins with the problems ML teams face: connecting to Kafka and other messaging systems, learning data-engineering frameworks, and scaling pipelines when event volume changes. Numaflow models a workload as a pipeline of containerized vertices connected by edges. Built-in sources and sinks handle data movement, while engineers focus on processing functions, model inference, or aggregations. The platform supports multiple languages and allows different components in one pipeline to use different languages. Yayi shows sentiment analysis, model comparison, and an Intuit anomaly-detection pipeline. Numaflow uses upstream pending events and component processing rates to scale individual stages. It also supports complex graphs, joins, conditional forwarding, reprocessing, custom sources and sinks, and event-driven training triggers. The talk ends with community examples and questions about production use and zero-downtime updates.

## Key ideas
### Event-driven systems let teams process incoming data asynchronously and at real-time scale
[01:58](https://www.youtube.com/watch?v=-EAexlXbN1I&t=118s)
Yayi describes event processing as receiving events and acting on them. Events can come from sensors, mouse clicks, or other applications. A system may analyze them, make predictions, and send results downstream. Event-driven designs support real-time processing and asynchronous work, which is useful when inference takes time. They also let teams scale components independently and replay events when needed. He gives examples including real-time recommendations from clicks and search history, actions based on IoT data, dynamic pricing, real-time analytics, and fraud detection.

### ML teams struggle with framework knowledge, integrations, and operational scaling
[04:55](https://www.youtube.com/watch?v=-EAexlXbN1I&t=295s)
Yayi says ML engineers often want to write event-processing logic in Python instead of learning large Java data-engineering frameworks. They also want to focus on processing and inference rather than connecting separately to Kafka, Pulsar, SQS, Kinesis, and downstream systems. Scaling creates another problem. Kubernetes autoscaling commonly uses CPU and memory, while streaming workloads also depend on Kafka consumer lag and the number of pending events. Platform teams then have to debug and maintain complex real-time infrastructure.

### Numaflow separates application logic from event sources, sinks, and infrastructure
[06:27](https://www.youtube.com/watch?v=-EAexlXbN1I&t=387s)
Numaflow aims to abstract infrastructure, separate sources and sinks from processing, and make pipelines scalable, reliable, and secure. Yayi models a pipeline as vertices connected by edges, with each vertex running as a container. In a sentiment-analysis example, an HTTP source provides text, a container runs a Hugging Face model, and a sink receives the result. Engineers write the processing function without needing to know whether the data came from Kafka, HTTP, Pulsar, or SQS.

### A small processing function can handle inference while Numaflow moves the data
[11:09](https://www.youtube.com/watch?v=-EAexlXbN1I&t=669s)
The example Python function receives messages, calls a sentiment-analysis model, and forwards the result. The developer does not configure the source connection inside that function. Yayi says the same pattern can support map operations, reduce operations, aggregations, or model serving. A YAML pipeline specification then defines the input source, the inference container, and the output sink. This leaves the team responsible for its processing logic while Numaflow handles movement through the pipeline.

### Pipeline graphs can compare models and combine preprocessing, inference, and training triggers
[12:51](https://www.youtube.com/watch?v=-EAexlXbN1I&t=771s)
Yayi shows a text-summarization pipeline that sends events to multiple model vertices for comparison. The models include different Hugging Face models, and their results are written to a log sink. He then shows a more complex Intuit anomaly-detection pipeline with preprocessing, inference, postprocessing, and multiple sinks. The same pipeline also includes a trainer vertex that triggers training workflows. Numaflow supports multiple sources, conditional forwarding, joins, reprocessing, and dynamic values passed into user-defined functions.

### Autoscaling follows streaming pressure and processing capacity at each stage
[14:41](https://www.youtube.com/watch?v=-EAexlXbN1I&t=881s)
Numaflow provisions separate pods for pipeline components and scales each stage individually. Yayi says it considers the current processing rate and the number of upstream events, then scales components so events do not build up as a backlog. The platform can scale from zero upward based on throughput and function processing rate. If a downstream database is unavailable, the pipeline can be scaled back to zero or write to a fallback sink. This approach considers streaming-specific signals instead of relying only on general resource usage.

### A single pipeline can use several programming languages
[15:41](https://www.youtube.com/watch?v=-EAexlXbN1I&t=941s)
Yayi presents a production anomaly-detection pipeline where inference uses Python, while preprocessing components use Java and Go. Numaflow provides SDKs for Java, Python, Go, and Rust. This lets teams choose a language for each component instead of forcing every part of a pipeline into one language. The platform also provides APIs for custom sources and sinks, including integrations that community users have built for systems such as ZeroMQ and RabbitMQ.

### Numaflow is already used for production and community workloads beyond ML
[18:58](https://www.youtube.com/watch?v=-EAexlXbN1I&t=1138s)
Yayi says Intuit completed a 1.0 release at the previous KubeCon and had used Numaflow in production at scale for more than a year. He names community uses involving audio processing, radio-frequency signals, IoT data, fraud detection, and other real-time processing. He says the platform can support high-throughput pipelines, including one processing close to 15,000 transactions per second, and can run on edge devices. A later answer explains that zero-downtime DAG updates were still under development.

## Notable quotes
- "Numaflow is a completely open source project that is developed by Intuit." (06:55)
- "All you need to do is you just need to implement this method where you are receiving bunch of messages." (11:09)
- "We autoscale each of those individually so that there is no backlog of events." (15:01)
- "It gives that flexibility to come together and then do any kind of processing that you have individually as a team." (16:00)
- "We have gone, we have done a 1.0 last KubeCon. It's been more than a year that we have been using it in production at scale." (20:22)

## Tools & references mentioned
- Intuit
- Numaflow
- Kubernetes
- Argo Workflows
- Kubeflow
- Kafka
- Apache Pulsar
- SQS
- Kinesis
- Hugging Face
- Python
- Java
- Go
- Rust
- ZeroMQ
- RabbitMQ
- KubeCon

## Who should watch
- You are building streaming inference or event-processing workloads and want model code to stay separate from messaging infrastructure.
- Your ML team uses Python while platform requirements involve Kafka, Kubernetes, autoscaling, and several downstream systems.
- You need to compare models, join streams, trigger training from events, or run processing on edge devices.

## Editor's note

Sri Harsha Yayi says ML engineers want to write event-processing logic in Python without learning large Java data-engineering frameworks or wiring each pipeline to Kafka and other systems. ZenML lets teams write workflows as Python steps while a configured stack handles the orchestrator and artifact store, so the same pipeline code can run across different infrastructure without being rewritten.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Real-Time Processing with Apache Flink, Kafka, and Pinot](https://mlopstalks.com/talks/real-time-processing-with-apache-flink-kafka-and-pinot) (Jacob Tsafatinos, Elemy, 53:41)
- [Metaflow: Supercharging Our Data Scientist Productivity](https://mlopstalks.com/talks/metaflow-supercharging-our-data-scientist-productivity) (Ravi Kiran Chirravuri, Netflix, 1:00:30)
- [Kubeflow vs MLflow](https://mlopstalks.com/talks/kubeflow-vs-mlflow) (Byron Allen, Servian, 54:57)
- [Packaging MLOps Tech Neatly for Engineers and Non-engineers](https://mlopstalks.com/talks/packaging-mlops-tech-neatly-for-engineers-and-non-engineers) (Jukka Remes, Haaga-Helia University of Applied Sciences, 8wave AI, 55:31)
- [Human-centric ML Infrastructure: A Netflix Original](https://mlopstalks.com/talks/human-centric-ml-infrastructure-a-netflix-original) (Savin Goyal, Netflix, 56:07)