# Streaming Machine Learning with Apache Kafka and Tiered Storage

Kai Waehner, Confluent | MLOps Meetup | Episode 34 | 52:50
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=Ur969-WX1BY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/streaming-machine-learning-with-apache-kafka-and-tiered-storage
Published: 2020-09-18
Tags: data-engineering, data-pipelines, model-serving, monitoring

## TL;DR
- Apache Kafka combines messaging, durable event storage, data integration, and stream processing, so teams can use it as the central event system for many applications.
- Tiered storage moves older Kafka data to object storage while keeping the Kafka API unchanged, making long-term retention cheaper and reducing broker rebalancing to the local hot set.
- Kafka can connect real-time machine learning training, scoring, monitoring, and replayable data with other systems, although batch processing and some model-serving features still need separate tools.

## Summary
Kai Waehner explains why Apache Kafka is more than a messaging system. Kafka stores ordered, timestamped events, lets producers and consumers operate independently, and supports retention policies that range from hours to years. Tiered storage moves older data from local broker disks to object storage, which makes long-term retention cheaper and limits rebalancing to the local hot set. This lets data scientists replay historical events directly from Kafka for analysis or model training. In a connected-car example, Kafka receives sensor data, stream processing prepares it, TensorFlow trains a model, and a Kafka application performs real-time scoring. Kai argues that embedding scoring in the Kafka application can avoid the latency and failure-handling problems of a separate HTTP model server for high-volume use cases. He also describes Kafka-based monitoring, reproducibility through immutable event logs, and changing model weights through Kafka topics. He is clear that Kafka does not replace every database or batch engine.

## Key ideas
### Kafka decouples producers, consumers, and retention
[02:22](https://www.youtube.com/watch?v=Ur969-WX1BY&t=142s)
Kai describes Kafka as messaging, storage, and a back-pressure system. Producers can keep writing even when consumers are offline or falling behind. Each topic can have its own retention period, such as two hours for logs or up to ten years for customer transactions. Because events remain ordered and timestamped, a consumer can later reprocess a selected period or replay data for analytics and model training. Kafka Connect links the event stream to legacy databases and mainframes, as well as systems such as HDFS, S3, MongoDB, and MQTT. Kafka Streams and KSQLDB provide processing on top of the stored events.

### Kafka can replace a separate database for some event workloads
[05:01](https://www.youtube.com/watch?v=Ur969-WX1BY&t=301s)
Kafka stores append-only events in guaranteed order, with timestamps that let consumers select a particular month or another slice of the log. Kai says this can be enough for some use cases, but Kafka does not provide native ANSI SQL and does not replace Oracle, Elasticsearch, or MongoDB for their specific workloads. A common pattern is to use Kafka as the central event system and build materialized views in MongoDB, S3, or a time-series database. Each downstream system consumes the events it needs and stores its own view.

### Kafka reduces a railway information system's collection of components
[10:25](https://www.youtube.com/watch?v=Ur969-WX1BY&t=625s)
Kai uses Deutsche Bahn as an example of why Kafka is different from a messaging queue. The railway needed real-time information, but station displays could only receive CSV files every ten minutes, while legacy systems also required database integration and processing. A RabbitMQ-based design grew to include storage, caching, processing, and integration frameworks. Deutsche Bahn re-engineered the system with Kafka, which provided messaging, storage, data integration, and processing in one distributed framework. The example was being rolled out across German stations at the time of the talk.

### Tiered storage makes long-term Kafka retention practical
[17:55](https://www.youtube.com/watch?v=Ur969-WX1BY&t=1075s)
Kafka brokers normally use attached disks for both processing and storage. That works when data expires after hours or days, but large retained datasets make disks expensive and make recovery slow when a broker fails or is replaced. With tiered storage, brokers keep a local hot set while older data moves to an object store such as Amazon S3 or Google Cloud Storage. Applications continue using the same Kafka API. Teams can choose how long each topic stays local before being offloaded. Kai says customers were already storing petabytes through tiered storage, and that rebalancing then affects only the local hot set rather than the full retained dataset.

### Kafka can feed machine learning directly from the event stream
[26:00](https://www.youtube.com/watch?v=Ur969-WX1BY&t=1560s)
In the connected-car example, cars send sensor data through MQTT into Kafka. Stream processing performs filtering, aggregation, and other preparation, while a Python application uses the Kafka and TensorFlow libraries to consume events and train a model. Training remains batch in most cases, although online training methods such as online clustering can work in real time. The trained model is placed in cloud object storage. A separate Kafka Streams application loads the model and scores events in real time, with Kafka providing ordering, availability, and delivery guarantees.

### Embedding scoring in Kafka trades model-server features for event-level performance
[29:00](https://www.youtube.com/watch?v=Ur969-WX1BY&t=1740s)
Kai compares embedding a model in a Kafka application with calling a model server over HTTP. Model servers offer features such as versioning and parameter changes out of the box. An embedded model can be a better fit when connected cars produce very high event volumes, low latency matters, or connectivity can be intermittent. Customers can send new model metadata or weights through a Kafka topic, allowing the scoring application to update without downtime. Regulated environments may require parallel versions and A/B testing so teams can prove which model handled each event.

### The Kafka event log supports replay, monitoring, and explanation
[31:20](https://www.youtube.com/watch?v=Ur969-WX1BY&t=1880s)
Because Kafka events are appended to a log and are not changed afterward, teams can replay the same input for different models and compare their outputs. Kai also describes using Kafka topics for monitoring. Technical metrics such as uptime and latency can flow to Prometheus and Grafana, while other consumers can write to TimescaleDB or InfluxDB. Business metrics, including model accuracy, require different monitoring choices. The same stored event history can support regulatory reporting and help explain what happened in a transaction or model workflow.

### Kafka complements batch engines and other infrastructure
[40:41](https://www.youtube.com/watch?v=Ur969-WX1BY&t=2441s)
Kai says Kafka processes messages in batches internally for performance, even when the application appears real time. It is not designed for large MapReduce-style batch jobs with shuffling, where Spark remains a better consumer. Kafka also complements systems such as MongoDB, rather than replacing them in every case. He describes uses ranging from mainframe integration and industrial IoT to transactional systems for orders, fraud detection, payments, and trading. In these workloads, zero downtime, zero data loss, and exactly-once semantics can matter more than very large data volumes.

## Notable quotes
- Kai Waehner: "Kafka is messaging and storage in the middle at scale highly available, but also it provides tools for data integration with Kafka Connect." (03:17)
- Kai Waehner: "Kafka provides all of this in one single component because again Kafka is messaging its storage, its data integration and its data processing, all of that in one framework at scale in real time." (13:44)
- Kai Waehner: "The great news is for the application nothing changes, the Kafka API is exactly the same like before, no single code change." (20:39)
- Kai Waehner: "The model training in most cases is still batch because that's how machine learning works today." (27:22)
- Kai Waehner: "If you want to build monitoring in the end it's just one more consumer and one application on top of that." (39:46)

## Tools & references mentioned
- Apache Kafka
- Kafka Connect
- Kafka Streams
- KSQLDB
- RabbitMQ
- HDFS
- Amazon S3
- Google Cloud Storage
- MongoDB
- MQTT
- TensorFlow
- Spark
- Flink
- Prometheus
- Grafana
- TimescaleDB
- InfluxDB
- DataRobot
- Kubernetes
- Confluent Platform
- Deutsche Bahn
- Uber
- Audi
- LinkedIn
- Cloudera
- Hortonworks
- Oracle
- Elasticsearch
- Redis
- Envoy
- Istio
- Seldon
- TF Serving
- Google Cloud Platform
- Pure Storage
- MinIO
- Hadoop

## Who should watch
- You are deciding whether Kafka should be only an ingestion layer or the central event system for real-time applications.
- Your machine learning pipeline needs replayable historical events, direct stream consumption, or high-volume online scoring.
- You are evaluating tiered storage and want to understand its effect on Kafka cost, retention, and broker recovery.

## Editor's note

Kai Waehner describes how immutable Kafka events let teams replay the same input for different models and compare their outputs. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a model or artifact can be traced back to the data and code that produced it. That gives replay results a record of their origin.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Real-Time Processing with Apache Flink, Kafka, and Pinot](https://mlopstalks.com/talks/real-time-processing-with-apache-flink-kafka-and-pinot) (Jacob Tsafatinos, Elemy, 53:41)
- [Streaming Ecosystem Complexities and Cost Management](https://mlopstalks.com/talks/streaming-ecosystem-complexities-and-cost-management) (Rohit Agrawal, Tecton, 47:39)
- [The Future of Software Architecture for GenAI: Real-Time Data Streaming](https://mlopstalks.com/talks/the-future-of-software-architecture-for-genai-real-time-data-streaming) (, 12:19)
- [Databricks Model Serving V2](https://mlopstalks.com/talks/databricks-model-serving-v2) (Rafael Pierre, Databricks, 43:17)
- [Machine Learning at Reasonable Scale](https://mlopstalks.com/talks/machine-learning-at-reasonable-scale) (Jacopo Tagliabue, Coveo, 1:04:32)