Training orchestration lets teams run compute-intensive training remotely on scalable Kubernetes infrastructure, including GPU-enabled clusters.
2
Model monitoring compares production data with the training baseline, detects drift, and can alert teams when predictions may no longer be meaningful.
3
A practical ML platform needs tests for serving code, training data, preprocessing logic, and model behavior before and after deployment.
Summary
Alon Gubkin continues the platform built in the first workshop, where DVC handled data versioning, MLflow tracked experiments, and FastAPI served models. He adds model monitoring with Aporia and training orchestration with Flyte. The monitoring integration logs the model schema, model version, training set, production inputs, and predictions. A model can then compare production distributions with its training baseline, detect drift, and send alerts. Alon demonstrates this with an Iris model whose predictions remain available for values far outside the training range. For training orchestration, he breaks preprocessing, training, and evaluation into Flyte tasks. Cached preprocessing can be reused across multiple training runs, and Kubernetes autoscaling can reduce idle compute costs. The session also covers Pulumi stacks for separate development, staging, and production environments. In the questions, Alon recommends unit tests for serving logic and preprocessing, data checks for schemas and value ranges, and reviewing both processed data and code during pull requests.
The platform adds orchestration and monitoring to the first workshop's foundation
The first session produced a basic ML infrastructure for data versioning, experiment management, packaging, and model serving. Alon used DVC, MLflow, and FastAPI for those parts. The community survey asked what to add next, and training orchestration and model monitoring received the most interest. The second session therefore extends the platform in those two areas. Alon keeps the earlier split between shared infrastructure and a model template that data scientists can clone for new projects.
Training orchestration moves expensive jobs from local machines to scalable Kubernetes clusters
Alon says orchestration matters when training is compute-intensive or takes too long to run locally. A system such as Flyte can send training code to a remote Kubernetes cluster, which can scale as needed and include GPUs for deep learning. He installs the orchestration system alongside the platform's existing infrastructure. The goal is to let data scientists run larger jobs without manually managing the machines where those jobs execute.
Monitoring needs a training baseline because production data can stop matching the training data
A model trained several months ago may no longer represent current reality. Alon demonstrates the problem with an Iris model whose training features mostly fall between one and six. The serving endpoint still returns a prediction when given values such as 100 or 200, even though those inputs are unrelated to the training data. The output is mathematically produced but may be meaningless. A monitoring system can detect this mismatch in production.
Logging the schema and training set gives monitoring the information it needs
The Aporia integration starts by installing and importing the package, then logging the model schema. Alon calls the schema important because a third-party source might change a feature from numeric to string and break the model. The integration also records the model version, using the MLflow run ID, and logs the training set. For very large training sets, the system sends the aggregations needed for drift detection instead of moving the entire dataset.
Production predictions need identifiers so later ground truth can be joined to them
When serving the model, Alon logs each input and output to Aporia and associates the prediction with a model and version. Each prediction also needs an ID. In a recommendation example, the user may click the recommended item, ignore it, or choose something else later. The ID lets the team add the eventual ground truth and calculate production metrics such as accuracy and F1 score. A sudden production drop could then trigger investigation, data collection, and retraining.
Drift monitors should be customized to each model's data and alert threshold
Alon creates a data drift monitor using the training data as the baseline. He chooses which features to inspect, sets a daily monitoring window, and adjusts the threshold based on historical drift scores. The interface shows the training distribution alongside the recent production distribution, making it possible to see where they differ. He also configures a high-severity alert and limits how many notifications can arrive in a day. Alon says a real platform could create basic monitors through an API, while data scientists customize monitors for individual models.
Flyte tasks make repeated training cheaper by caching reusable preprocessing
Alon describes Flyte as a workflow automation platform that is useful for training orchestration. A training pipeline can be split into tasks for preprocessing, training, and post-processing or evaluation. If several hyperparameter runs use the same preprocessing, that task can be cached. The later runs can reuse the cached result instead of processing a very large dataset again. He presents this as especially useful when training data reaches terabyte or petabyte scale.
Tests should cover serving behavior, data quality, preprocessing, and model outputs
In response to Demetrios Brinkmann, Alon separates testing into several areas. Unit tests can send requests to the FastAPI server and check the response for different inputs. Tests can also run through a training or test dataset and compare predictions with expected values. Data tests should check the schema and whether values are in range. Alon warns that preprocessing branches can silently create missing features, which may produce an invalid model during automatic training. Reviewing distributions and the preprocessing code during a pull request helps catch this.
