Google's MLOps model treats machine learning development and operations as one system, with automation, monitoring, and repeatable pipelines across the lifecycle.
2
Continuous training is useful when production data changes, but many teams do not need to retrain models automatically or frequently.
3
A mature MLOps system can still include manual work when the problem is experimental or the cost of automating it is greater than its benefit.
Summary
Demetrios Brinkmann and David Aponte discuss Google's article on continuous delivery and automation pipelines for machine learning. They explain Google's view of MLOps as both an engineering culture and a set of practices that connect development with operations. The proposed system covers data extraction, feature engineering, model training, deployment, monitoring, and retraining, with metadata linking each pipeline run to its resulting model. David describes how a repository, CI/CD system, and pipeline framework such as Kubeflow could package these stages into reusable components. The conversation also separates continuous integration, continuous delivery, and continuous training. The hosts are careful about the limits of Google's maturity model. Automation can reduce manual errors and make repeatability easier, but it can also add cost and complexity while the data science work is still exploratory. Continuous training makes sense when data or user behavior changes, though it is not required for every model.
Automation should follow a business need instead of arriving first
Demetrios recalls Charles Martin's advice to establish a business case, get the system working manually, and automate it after repeated manual runs become painful. Phil Winder's MLOps hierarchy of needs places automation at the top rather than at the foundation. The hosts question the assumption that a maturity level of zero is automatically bad. David adds that a fully automated system can still produce poor machine learning results, because software correctness does not guarantee that the model or data is appropriate.
David quotes Google's definition of MLOps as "a machine learning engineering culture and a practice" that unifies machine learning system development with operations. The cultural side concerns how teams view and interact with the system. The practical side includes repeatable processes, automation, and monitoring. In Google's mature version, the whole process can run from beginning to end in a predictable and controlled way instead of relying on ad hoc human actions.
The pipeline includes data work before model training
Google's lifecycle starts with experimentation and development, continues through training, and ends with serving. David says the training system should include data extraction, feature engineering, and validation, rather than focusing only on the model. Teams may be trying new data sources, signals, transformations, or fresh datasets. They may also be changing hyperparameters and model architectures. The proposed goal is reliable and repeatable training across both data development and model development.
Configuration should change without rewriting the experiment
David describes an orchestrated experiment whose source code stays reusable while configuration changes select features, transformations, data sources, model parameters, learning rates, or compute resources. The same approach can describe one model or a larger model system, such as an ensemble. This creates a repeatable unit of work and allows teams to compare runs while preserving the settings that produced them.
A repository can connect CI/CD with containerized pipeline components
David gives an example of a model repository containing feature generation, training, and serving components. A CI/CD system such as Jenkins, GitLab, or GitHub Actions can run tests and deploy changes. With Kubeflow, each stage can become a containerized pipeline component. The feature component prepares inputs, the training component creates a model, and the serving component loads the model and produces predictions. Parameters can be passed through a YAML file, while source commits and image tags provide version history.
Lineage links a trained model to the pipeline that produced it
The proposed system records metadata about each run, including who started it, how long it ran, which resources were used, and which data and model parameters were selected. David says the pipeline artifact and model artifact should remain linked. That connection allows a team to identify the pipeline version, data preparation, validation, training configuration, and evaluation associated with a deployed model. Metrics can then connect technical behavior with business measures.
Continuous training responds to changing production conditions
Continuous training means automatically retraining and serving models. David gives changing feature distributions, user preferences, marketing campaigns, competitor moves, weather, news, location, and device changes as situations that may make a model stale. A production system can monitor performance and data, detect drift, rerun feature generation and training, and redeploy a new model. This only fits systems where the world changes enough to justify that process.
Manual work remains appropriate for experimental machine learning
The hosts stress that continuous retraining is not used heavily in every company, even when the capability exists. David says premature automation can create overhead and over-engineering while teams are still trying to discover what works. A system can be reliable and reproducible without fully automated retraining. Documentation of manual steps, dashboards, and repeatable procedures can reduce errors when automation is not the right priority.
CI, CD, and CT describe different parts of the lifecycle
Continuous integration merges small, frequent code changes and validates them with tests for code, data, schemas, model configuration, and related components. Continuous delivery keeps a system ready for release and can deploy a training pipeline or prediction service after validation. Continuous training reruns training and serving when the data or environment changes. The hosts also discuss canary releases, silent deployments, feature flags, and A/B tests as ways to evaluate model changes in production.
