# Building an ML Platform from Scratch: Live Coding Session - Part 2

Alon Gubkin, Aporia | MLOps Meetup | Episode 74 | 1:12:53
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=C2y72n2oyqs
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-an-ml-platform-from-scratch-live-coding-session-part-2
Published: 2021-08-09
Tags: drift, monitoring, orchestration, testing

## TL;DR
- Training orchestration lets teams run compute-intensive training remotely on scalable Kubernetes infrastructure, including GPU-enabled clusters.
- Model monitoring compares production data with the training baseline, detects drift, and can alert teams when predictions may no longer be meaningful.
- A practical ML platform needs tests for serving code, training data, preprocessing logic, and model behavior before and after deployment.

## Summary
Alon Gubkin continues the platform built in the first workshop, where DVC handled data versioning, MLflow tracked experiments, and FastAPI served models. He adds model monitoring with Aporia and training orchestration with Flyte. The monitoring integration logs the model schema, model version, training set, production inputs, and predictions. A model can then compare production distributions with its training baseline, detect drift, and send alerts. Alon demonstrates this with an Iris model whose predictions remain available for values far outside the training range. For training orchestration, he breaks preprocessing, training, and evaluation into Flyte tasks. Cached preprocessing can be reused across multiple training runs, and Kubernetes autoscaling can reduce idle compute costs. The session also covers Pulumi stacks for separate development, staging, and production environments. In the questions, Alon recommends unit tests for serving logic and preprocessing, data checks for schemas and value ranges, and reviewing both processed data and code during pull requests.

## Key ideas
### The platform adds orchestration and monitoring to the first workshop's foundation
[04:06](https://www.youtube.com/watch?v=C2y72n2oyqs&t=246s)
The first session produced a basic ML infrastructure for data versioning, experiment management, packaging, and model serving. Alon used DVC, MLflow, and FastAPI for those parts. The community survey asked what to add next, and training orchestration and model monitoring received the most interest. The second session therefore extends the platform in those two areas. Alon keeps the earlier split between shared infrastructure and a model template that data scientists can clone for new projects.

### Training orchestration moves expensive jobs from local machines to scalable Kubernetes clusters
[04:43](https://www.youtube.com/watch?v=C2y72n2oyqs&t=283s)
Alon says orchestration matters when training is compute-intensive or takes too long to run locally. A system such as Flyte can send training code to a remote Kubernetes cluster, which can scale as needed and include GPUs for deep learning. He installs the orchestration system alongside the platform's existing infrastructure. The goal is to let data scientists run larger jobs without manually managing the machines where those jobs execute.

### Monitoring needs a training baseline because production data can stop matching the training data
[05:53](https://www.youtube.com/watch?v=C2y72n2oyqs&t=353s)
A model trained several months ago may no longer represent current reality. Alon demonstrates the problem with an Iris model whose training features mostly fall between one and six. The serving endpoint still returns a prediction when given values such as 100 or 200, even though those inputs are unrelated to the training data. The output is mathematically produced but may be meaningless. A monitoring system can detect this mismatch in production.

### Logging the schema and training set gives monitoring the information it needs
[19:54](https://www.youtube.com/watch?v=C2y72n2oyqs&t=1194s)
The Aporia integration starts by installing and importing the package, then logging the model schema. Alon calls the schema important because a third-party source might change a feature from numeric to string and break the model. The integration also records the model version, using the MLflow run ID, and logs the training set. For very large training sets, the system sends the aggregations needed for drift detection instead of moving the entire dataset.

### Production predictions need identifiers so later ground truth can be joined to them
[30:09](https://www.youtube.com/watch?v=C2y72n2oyqs&t=1809s)
When serving the model, Alon logs each input and output to Aporia and associates the prediction with a model and version. Each prediction also needs an ID. In a recommendation example, the user may click the recommended item, ignore it, or choose something else later. The ID lets the team add the eventual ground truth and calculate production metrics such as accuracy and F1 score. A sudden production drop could then trigger investigation, data collection, and retraining.

### Drift monitors should be customized to each model's data and alert threshold
[34:33](https://www.youtube.com/watch?v=C2y72n2oyqs&t=2073s)
Alon creates a data drift monitor using the training data as the baseline. He chooses which features to inspect, sets a daily monitoring window, and adjusts the threshold based on historical drift scores. The interface shows the training distribution alongside the recent production distribution, making it possible to see where they differ. He also configures a high-severity alert and limits how many notifications can arrive in a day. Alon says a real platform could create basic monitors through an API, while data scientists customize monitors for individual models.

### Flyte tasks make repeated training cheaper by caching reusable preprocessing
[41:00](https://www.youtube.com/watch?v=C2y72n2oyqs&t=2460s)
Alon describes Flyte as a workflow automation platform that is useful for training orchestration. A training pipeline can be split into tasks for preprocessing, training, and post-processing or evaluation. If several hyperparameter runs use the same preprocessing, that task can be cached. The later runs can reuse the cached result instead of processing a very large dataset again. He presents this as especially useful when training data reaches terabyte or petabyte scale.

### Tests should cover serving behavior, data quality, preprocessing, and model outputs
[52:49](https://www.youtube.com/watch?v=C2y72n2oyqs&t=3169s)
In response to Demetrios Brinkmann, Alon separates testing into several areas. Unit tests can send requests to the FastAPI server and check the response for different inputs. Tests can also run through a training or test dataset and compare predictions with expected values. Data tests should check the schema and whether values are in range. Alon warns that preprocessing branches can silently create missing features, which may produce an invalid model during automatic training. Reviewing distributions and the preprocessing code during a pull request helps catch this.

## Notable quotes
- Alon Gubkin: "Training orchestration is important when your training is compute intensive or just takes a lot of time." (04:43)
- Alon Gubkin: "Logging the schema of the data is extremely important." (20:38)
- Alon Gubkin: "You always want to use the same pre-processing logic." (47:40)
- Alon Gubkin: "I would also write tests for my training data themselves." (1:02:30)

## Tools & references mentioned
- DVC
- MLflow
- FastAPI
- Kubernetes
- Pulumi
- Cookiecutter
- Flyte
- Lyft
- Aporia
- Traefik
- Amazon S3
- pytest
- pandas profiling

## Who should watch
- You are building an internal ML platform and need to add monitoring or remote training to an existing MLflow and Kubernetes setup.
- Your production models receive inputs that can drift away from their training data, and you need alerts tied to model versions and later ground truth.
- You review data science pull requests and want practical guidance on testing serving code, preprocessing, datasets, and model behavior.

## Editor's note

Alon Gubkin says compute-intensive training should move from local machines to scalable Kubernetes clusters, so data scientists do not have to manage the machines themselves. ZenML lets teams write workflows as Python steps and choose an orchestrator through configuration, so the same pipeline code can run on a laptop or Kubernetes. That gives this problem a direct path from local development to remote training.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Building an ML Platform from Scratch: Live Coding Session](https://mlopstalks.com/talks/building-an-ml-platform-from-scratch-live-coding-session) (Alon Gubkin, Aporia, 1:57:24)
- [Building an ML Platform: Insights, Community, and Advocacy](https://mlopstalks.com/talks/building-an-ml-platform-insights-community-and-advocacy) (Stephen Batifol, Wolt, 45:49)
- [How to Leverage ML Tooling Ecosystem](https://mlopstalks.com/talks/how-to-leverage-ml-tooling-ecosystem) (Mariya Davydova, Neu.ro, 55:57)
- [DevOps, Security, and Observability in ML](https://mlopstalks.com/talks/devops-security-and-observability-in-ml) (Luke Marsden, MLOps Consulting, 32:46)
- [The Future of ML and Data Platforms](https://mlopstalks.com/talks/the-future-of-ml-and-data-platforms) (Michael Del Balso, Tecton, 55:17)