# Building ML Blocks with Kubeflow Orchestration with Feature Store

Aniruddha Choudhury, Publicis Sapient | MLOps Meetup | Episode 72 | 1:26:03
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=UJLQgxr_Za0
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-ml-blocks-with-kubeflow-orchestration-with-feature-store
Published: 2021-07-23
Tags: feature-engineering, feature-stores, model-serving, monitoring, orchestration

## TL;DR
- A feature store gives data science teams a shared repository for engineered features that can be used consistently during training and serving.
- Feast can ingest both batch and streaming data, keep historical snapshots, and provide low-latency online feature retrieval through Redis.
- Kubeflow can connect feature retrieval, hyperparameter tuning with Katib, model training, serving with KFServing, and monitoring with Prometheus and Grafana in one pipeline.

## Summary
Aniruddha Choudhury walks through an end-to-end machine learning system built with Feast, Kubeflow, Kubernetes, Spark, Kafka, Redis, and Google Cloud. He starts with the problem of several data scientists independently collecting, transforming, and scaling the same data. A feature store provides a shared definition of features so teams can reuse them across projects and keep training data aligned with serving data. He then explains Feast's registry, transformations, offline storage, online serving, and monitoring. The demo uses taxi trip data for a classification task, registers entities and feature tables, ingests batch data into Google Cloud Storage, and moves data from offline storage to Redis for online access. The Kubeflow pipeline retrieves historical features, runs Katib hyperparameter trials, trains a model, saves it to a Google Cloud bucket, deploys it for prediction, and exposes metrics through Prometheus and Grafana. The example reaches 58 percent accuracy, while Choudhury says feature engineering could improve the result.

## Key ideas
### A feature store reduces duplicated feature work across data scientists
[05:08](https://www.youtube.com/watch?v=UJLQgxr_Za0&t=308s)
Choudhury says teams often spend time repeating data collection, analysis, and feature engineering. Different data scientists may apply different scaling or preprocessing because they have different experience and domain knowledge. This creates inconsistent training data and leaves ML or data engineers maintaining fragmented production datasets. A centralized feature store holds engineered features in a shared repository. Several data scientists can reuse the same features for regression or classification, while serving systems use the same definitions for predictions. Choudhury describes this as maintaining a single source of truth for training and serving.

### Feast separates feature storage from the rest of the ML platform
[10:10](https://www.youtube.com/watch?v=UJLQgxr_Za0&t=610s)
Feast is presented as an open-source feature store rather than a workflow scheduler, data warehouse, feature engineering tool, model serving product, or pipeline orchestrator. It does not replace Airflow, Kubeflow, Spark, pandas, or model serving systems. Its role is to manage feature definitions and provide a shared repository that teams can use across projects. Choudhury says Feast supports batch and streaming ingestion, feature reuse, metadata, and consistent access for training and serving. He also describes support for real-time feature access through cloud connectors and online storage.

### The feature store has separate layers for serving, transformation, monitoring, storage, and registry
[11:44](https://www.youtube.com/watch?v=UJLQgxr_Za0&t=704s)
Choudhury describes five parts of a modern feature store. The serving layer provides low-latency access for predictions. Transformation includes batch, streaming, and on-demand processing. Monitoring tracks measures such as latency, throughput, freshness, availability, and capacity. The registry stores feature definitions and metadata as a shared reference. Storage keeps historical snapshots in systems such as Google Cloud Storage, S3, Snowflake, Redshift, DynamoDB, Redis, or Cassandra. This structure lets the same engineered data support both historical training retrieval and fast online prediction.

### Feast keeps offline training data aligned with online prediction data
[14:55](https://www.youtube.com/watch?v=UJLQgxr_Za0&t=895s)
In the taxi classification example, the data contains driver IDs, timestamps, trip details, payment information, and fare values. Choudhury registers the driver ID as an entity and defines feature tables for trip and fare data. Historical data is saved as snapshots in a Google Cloud bucket, while online data is moved into Redis for fast retrieval. Feast can retrieve historical features with event timestamps for training and online features by entity ID for serving. The training and prediction paths therefore use the same registered feature definitions instead of separately maintained datasets.

### Batch and streaming ingestion can feed the same feature system
[25:41](https://www.youtube.com/watch?v=UJLQgxr_Za0&t=1541s)
The demo first loads transformed CSV data and sends it through a batch ingestion path backed by a Spark job on a Google Cloud Dataproc cluster. Feast stores the resulting historical data in a Google Cloud Storage bucket using Parquet snapshots. Choudhury then shows a streaming path using Kafka topics and a Spark job. The Kafka source includes an event timestamp, a creation timestamp, a broker address, a topic, and a schema. Rows are published as streaming events, and the job continues running so new data can reach online features. The two paths support offline retrieval for training and online retrieval for production predictions.

### The Kubeflow pipeline is built from separate containerized stages
[41:28](https://www.youtube.com/watch?v=UJLQgxr_Za0&t=2488s)
The pipeline is divided into feature retrieval, Katib hyperparameter tuning, final training, and serving. Each stage uses a Docker image with its Python code and dependencies. The feature retrieval stage reads entity IDs, timestamps, feature names, and storage locations, then asks Feast for historical features through a Spark job. It saves training and test artifacts to a persistent volume. The training stage reads those artifacts, builds a TensorFlow model, writes logs, saves metrics, and uploads model weights to a Google Cloud bucket. Kubeflow's pipeline graph connects these stages with dependency ordering.

### Katib searches model configurations before final training
[53:35](https://www.youtube.com/watch?v=UJLQgxr_Za0&t=3215s)
Choudhury configures Katib with a maximum of 24 trials, three parallel trials, and a maximum of three failed trials. The objective is to maximize accuracy, with Bayesian optimization selecting values for learning rate, optimizer, layers, units, and initialization settings. An early-stopping rule can stop trials when accuracy stops improving. Katib reads the training logs and displays the trial results in its interface. In the shown run, the model reaches 58 percent accuracy. The best parameters are passed to a final training stage, which trains the model again and saves it for serving.

### Monitoring covers both feature access and model endpoint traffic
[1:20:12](https://www.youtube.com/watch?v=UJLQgxr_Za0&t=4812s)
The deployed system exposes metrics through Prometheus and Grafana. Choudhury mentions feature-serving measures such as request counts, latency, throughput, memory, concurrency, and availability. He also shows endpoint traffic after sending prediction requests to the KFServing service. The dashboard can show request volume and the pressure on the endpoint. Kubernetes can scale the service, while Grafana provides a view of the health of the feature store and serving layer.

## Notable quotes
- Aniruddha Choudhury: "We are maintaining a centralized repository of feature store which we have scaled and as a feature engineer process data and we have published it into the feature store once that and it can be used across all the projects or across all the data scientists in our team." (09:28)
- Aniruddha Choudhury: "Feast is not about a workflow or scheduler like Airflow or Kubeflow." (16:30)
- Aniruddha Choudhury: "We are maintaining a single source of truth, a similar sort of data, a customer ID, date time, gender and all the data has been consistent in training and similarly the same data has been consistent for prediction also." (16:10)
- Aniruddha Choudhury: "The very last step is to monitor the results, to monitor my metrics." (1:21:20)

## Tools & references mentioned
- MLOps Community
- Publicis Sapient
- Docker
- Kubeflow
- Kubernetes
- Katib
- Kafka
- Prometheus
- Grafana
- Feast
- Tecton
- Google Cloud
- Google Cloud Storage
- Dataproc
- Spark
- Redis
- Airflow
- MLflow
- Metaflow
- Prefect
- PyTorch
- TensorFlow
- scikit-learn
- S3
- Snowflake
- Redshift
- DynamoDB
- Cassandra
- KFServing
- TensorFlow Serving
- BentoML

## Who should watch
- You are choosing between a feature store and a collection of ad hoc tables, notebooks, and preprocessing scripts for training and serving.
- Your team wants a concrete example of combining Feast with Kubeflow, Spark, Kafka, Redis, and Google Cloud.
- You need to connect hyperparameter tuning, model deployment, endpoint prediction, and operational metrics in a Kubernetes-based ML system.

## Editor's note

Aniruddha Choudhury shows how separate Kubeflow stages connect feature retrieval, tuning, training, and serving, while Feast keeps training and serving data aligned. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a trained model can be traced back to the data and code that produced it. The same pipeline code can run on different infrastructure through configuration.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [2 tools = 90% operational ML](https://mlopstalks.com/talks/2-tools-90-operational-ml) (Michael Del Balso, Tecton & Willem Pienaar, Feast & David Aronchick, Kubeflow, 56:39)
- [Feature Stores: An Essential Part of the ML Stack to Build Great Data](https://mlopstalks.com/talks/feature-stores-an-essential-part-of-the-ml-stack-to-build-great-data) (Kevin Stumpf, Tecton, 1:05:46)
- [Feast Feature Store Deep Dive](https://mlopstalks.com/talks/feast-feature-store-deep-dive) (Felix Wang, Tecton, 28:36)
- [Machine Learning Feature Store Panel Discussion](https://mlopstalks.com/talks/machine-learning-feature-store-panel-discussion) (Vishnu Rachakonda, Tesseract Health & Daniel Galinkin, iFood & Matias Dominguez, Rappi & Simarpal Khaira, Intuit, 1:05:16)
- [How Feature Stores Work](https://mlopstalks.com/talks/how-feature-stores-work) (Simba Khadder, Featureform, 30:33)