The machine learning process moves through extraction, analysis, preparation, training, evaluation, validation, serving, and monitoring, with repeated movement between stages during experimentation.
2
Maturity level zero uses mostly manual, script-driven work and deploys individual prediction services, while maturity level one automates and deploys the whole training pipeline.
3
Automation should follow a stable, repeated process. Teams should first monitor manual systems, learn where the recurring work is, and avoid building an elaborate system for a problem that may change.
Summary
Demetrios Brinkmann and David Ponte use the Google paper to explain the steps from raw data to a monitored prediction service. They stress that machine learning development is iterative. Data analysis and preparation overlap, feature choices depend on the model, and training and serving must use consistent schemas. Evaluation also needs a connection between model metrics and business outcomes. The discussion then compares two maturity levels. At level zero, data scientists work in notebooks or one-off scripts, operations and experimentation are separated, and only a trained model is deployed manually. At level one, an orchestrated pipeline validates data, prepares features, trains and validates models, records metadata, registers artifacts, serves the model, and can retrain after monitoring detects degradation. David explains why feature stores, reusable containerized components, metadata, and event-based triggers help. They also caution against over-engineering and say level two adds CI/CD for pipeline deployment.
The data science workflow moves back and forth instead of running as a fixed sequence
Demetrios describes data extraction, data analysis, and data preparation as the early parts of the machine learning process. David adds that analysis can depend on the intended algorithm. CatBoost can encode categorical and text features, while XGBoost may require separate one-hot encoding. A team may also try a baseline model or an unsupervised approach during analysis. Demetrios sums up the process as a "zigzag": people may prepare data while analysing it, then return to earlier decisions. That makes reproducibility difficult because experimentation needs to stay fast while preserving the exact steps and inputs that produced a result.
Training data and serving data need the same feature schema
During data preparation, the team splits data into training, validation, and test sets, cleans it, and performs feature engineering. David says the test set should resemble the live situation and that a model should generalize to new data. The feature format also needs to remain consistent at inference time. The number of columns and the schema used during training must match what the prediction service receives. David introduces the feature store as one way to help keep offline training data and online serving data aligned. He distinguishes large offline reads from low-latency online reads, where a separate database may be needed.
Model evaluation must connect model metrics to business outcomes
After training and hyperparameter tuning, teams evaluate models and validate whether they are suitable for deployment. Demetrios mentions accuracy, while David separates model metrics from business metrics. Metrics such as ROC, training loss, accuracy, and recall describe model behaviour, but a business may care about click-through rate or another outcome. David says low-level metrics can be gamed, and the link to business results can be hard to establish when the data is sparse or unreliable. Serving then exposes the validated model through an online endpoint, a batch system, or an embedded model on an edge device.
Monitoring is part of machine learning operations after deployment
The conversation treats monitoring as the main part of "day two operations." Once a model is serving predictions, the team needs to check whether it still behaves as expected, whether drift is occurring, and whether bugs have entered the system. David mentions SAM, an unsupervised approach for automatic model monitoring that can detect concept drift and trigger an action such as retraining. Demetrios recalls a recommender system that showed the same product for 18 days, which caused a large financial loss. The example explains why a deployed model needs observation rather than being left alone after release.
Maturity level zero keeps experimentation and operations largely separate
The first maturity diagram from the Google paper describes a manual, script-driven process. Data scientists and researchers try different ideas, often in notebooks or one-off scripts, while operations or infrastructure teams work separately. There is no continuous integration or continuous delivery, and releases of prediction services happen infrequently. Only the trained model is deployed to production. The experimental environment contains the manual work, while production contains the resulting prediction service. David and Demetrios also qualify the paper's criticism: manual release can be reasonable when a model changes rarely, such as a healthcare system that may keep one model for 18 months. The needed level of automation depends on the domain and release pattern.
Automation should be introduced after a repeated process becomes clear
David says automation makes sense when a team performs the same sequence repeatedly and understands the recipe. He warns against over-engineering a large system for a problem that may change. Demetrios and David compare this to building an elaborate bridge for a river that later moves. For a manual system, they recommend monitoring the production model, recording events, and using degradation or stale data as signals for new experiments. Teams can then retrain on recent data or revisit a corrupt data source. David also cautions that constant retraining is not automatically the answer. A model should generalize to new data, and repeated retraining may indicate a deeper problem.
Maturity level one deploys an automated pipeline instead of only a model
At maturity level one, the team automates data validation, data preparation, model training, and model validation in an orchestrated pipeline. The pipeline code is integrated into source control, and its output is a trained artifact that can enter a model registry and then a serving environment. Monitoring can trigger the entire pipeline again, including data extraction and preparation. This changes production from a manually released prediction service into an automated process that can reproduce the model. The diagram adds a feature store, metadata store, source repository, and source code. Demetrios describes the diagram as much more complex because the experimental and production environments now contain related pipeline stages.
Reusable components and recorded metadata make the pipeline reproducible
David describes an orchestrated experiment as a pipeline whose output can be stored as source code and promoted into production. Each component can be containerized and reused. Separate containers can carry different dependencies or languages, while isolated runtime environments make deployment more reproducible. Metadata records the pipeline version, data, model, parameters, and individual jobs. David says this lets a team find the artifact associated with a run, roll back, and reproduce the steps that created it. The feature store also needs metadata such as a feature's source, creation time, owner, and value range. Data receives the same operational care as model code because model quality depends on the data supplied to it.
Triggers turn monitoring and schedules into automated retraining paths
The level one pipeline can react to events or run on a schedule. A monitoring function may publish an event when a metric falls below a threshold, and another service can subscribe and trigger the pipeline. Demetrios and David describe this as a decoupled publisher and subscriber pattern. A scheduled batch system is another option, such as running a daily job and writing predictions to a database. The next maturity level is introduced as a response to a remaining problem: testing new ideas can still require manually deploying pipelines. The hosts say level two adds CI/CD to automate that pipeline deployment.
"A model should be able to generalize well. It should be able to perform well on new instances of data, not the things that it memorized or learned during the training time."David Ponte09:34
Who should watch
You are deciding whether a manual model release process is sufficient for a system that changes slowly, or whether repeated releases justify automation.
Your team has separate notebooks, training code, serving code, and production infrastructure, and you need to understand where consistency and reproducibility break down.
You want a practical explanation of feature stores, metadata, monitoring triggers, and whole-pipeline deployment before adopting CI/CD or continuous training.
