Numaflow lets teams define streaming workloads as Kubernetes-native pipelines without managing connections to each event source and sink.
2
Teams can write processing and inference logic in Python, Java, Go, or Rust while Numaflow handles data movement between pipeline components.
3
Numaflow scales pipeline components using upstream backlog and processing rate, rather than relying only on CPU and memory usage.
Summary
Sri Harsha Yayi introduces Numaflow, an open-source, Kubernetes-native platform from Intuit for real-time event processing and inference. He begins with the problems ML teams face: connecting to Kafka and other messaging systems, learning data-engineering frameworks, and scaling pipelines when event volume changes. Numaflow models a workload as a pipeline of containerized vertices connected by edges. Built-in sources and sinks handle data movement, while engineers focus on processing functions, model inference, or aggregations. The platform supports multiple languages and allows different components in one pipeline to use different languages. Yayi shows sentiment analysis, model comparison, and an Intuit anomaly-detection pipeline. Numaflow uses upstream pending events and component processing rates to scale individual stages. It also supports complex graphs, joins, conditional forwarding, reprocessing, custom sources and sinks, and event-driven training triggers. The talk ends with community examples and questions about production use and zero-downtime updates.
Event-driven systems let teams process incoming data asynchronously and at real-time scale
Yayi describes event processing as receiving events and acting on them. Events can come from sensors, mouse clicks, or other applications. A system may analyze them, make predictions, and send results downstream. Event-driven designs support real-time processing and asynchronous work, which is useful when inference takes time. They also let teams scale components independently and replay events when needed. He gives examples including real-time recommendations from clicks and search history, actions based on IoT data, dynamic pricing, real-time analytics, and fraud detection.
ML teams struggle with framework knowledge, integrations, and operational scaling
Yayi says ML engineers often want to write event-processing logic in Python instead of learning large Java data-engineering frameworks. They also want to focus on processing and inference rather than connecting separately to Kafka, Pulsar, SQS, Kinesis, and downstream systems. Scaling creates another problem. Kubernetes autoscaling commonly uses CPU and memory, while streaming workloads also depend on Kafka consumer lag and the number of pending events. Platform teams then have to debug and maintain complex real-time infrastructure.
Numaflow separates application logic from event sources, sinks, and infrastructure
Numaflow aims to abstract infrastructure, separate sources and sinks from processing, and make pipelines scalable, reliable, and secure. Yayi models a pipeline as vertices connected by edges, with each vertex running as a container. In a sentiment-analysis example, an HTTP source provides text, a container runs a Hugging Face model, and a sink receives the result. Engineers write the processing function without needing to know whether the data came from Kafka, HTTP, Pulsar, or SQS.
A small processing function can handle inference while Numaflow moves the data
The example Python function receives messages, calls a sentiment-analysis model, and forwards the result. The developer does not configure the source connection inside that function. Yayi says the same pattern can support map operations, reduce operations, aggregations, or model serving. A YAML pipeline specification then defines the input source, the inference container, and the output sink. This leaves the team responsible for its processing logic while Numaflow handles movement through the pipeline.
Pipeline graphs can compare models and combine preprocessing, inference, and training triggers
Yayi shows a text-summarization pipeline that sends events to multiple model vertices for comparison. The models include different Hugging Face models, and their results are written to a log sink. He then shows a more complex Intuit anomaly-detection pipeline with preprocessing, inference, postprocessing, and multiple sinks. The same pipeline also includes a trainer vertex that triggers training workflows. Numaflow supports multiple sources, conditional forwarding, joins, reprocessing, and dynamic values passed into user-defined functions.
Autoscaling follows streaming pressure and processing capacity at each stage
Numaflow provisions separate pods for pipeline components and scales each stage individually. Yayi says it considers the current processing rate and the number of upstream events, then scales components so events do not build up as a backlog. The platform can scale from zero upward based on throughput and function processing rate. If a downstream database is unavailable, the pipeline can be scaled back to zero or write to a fallback sink. This approach considers streaming-specific signals instead of relying only on general resource usage.
A single pipeline can use several programming languages
Yayi presents a production anomaly-detection pipeline where inference uses Python, while preprocessing components use Java and Go. Numaflow provides SDKs for Java, Python, Go, and Rust. This lets teams choose a language for each component instead of forcing every part of a pipeline into one language. The platform also provides APIs for custom sources and sinks, including integrations that community users have built for systems such as ZeroMQ and RabbitMQ.
Numaflow is already used for production and community workloads beyond ML
Yayi says Intuit completed a 1.0 release at the previous KubeCon and had used Numaflow in production at scale for more than a year. He names community uses involving audio processing, radio-frequency signals, IoT data, fraud detection, and other real-time processing. He says the platform can support high-throughput pipelines, including one processing close to 15,000 transactions per second, and can run on edge devices. A later answer explains that zero-downtime DAG updates were still under development.
