MLOps combines software engineering, DevOps, and machine learning across the full lifecycle from data work to production models.
2
The first problem to solve is getting models into production; teams also need reproducibility, accountability, asynchronous collaboration, and continuous development and monitoring.
3
Machine learning collaboration needs versioning for data, runs, parameters, metrics, environments, and models, using an approach similar to GitHub branches and pull requests.
Summary
Luke Marsden introduces MLOps as the intersection of software engineering, DevOps, and machine learning. He describes machine learning teams that can train models on laptops but cannot deploy them, lose track of training data and parameters, or discover failures after deployment because their models are not monitored. His proposed tests for an MLOps process cover reproducibility, accountability, asynchronous collaboration, and continuous development. The talk explains why machine learning has more moving parts than ordinary software: data, data-processing runs, code, parameters, environments, model runs, and serialized model artifacts all affect the result. Marsden then applies software collaboration patterns to distributed data science teams. He demonstrates Dotscience concepts including runs, provenance, forks, pull requests, data versioning, and metric comparison, although the live update and merge demonstration does not complete. In the question period, he also places infrastructure provisioning, flexible compute, and multi-cloud or on-premises resources within an MLOps platform.
MLOps joins software engineering, DevOps, and machine learning
Luke Marsden defines MLOps as the intersection of software engineering, DevOps, and machine learning. Software engineering creates software, DevOps deploys and operates it, and machine learning uses data and mathematics to train models that make predictions. He argues that machine learning is moving from research into production, so these disciplines need to converge. Teams need to build and deploy models with the level of rigor and reliability expected from modern software and DevOps practices. MLOps therefore covers the end-to-end process rather than only the operation of already-deployed models.
The first operational problem is getting models into production
Marsden says many companies can develop models on laptops but cannot get them deployed into production. This raises time to value, and a deployment that takes three or six months may use stale data by the time it goes live. Other failures follow later. Models may run without monitoring, teams may waste time making development environments compatible, and data scientists may copy datasets between machines. Teams also track code, data, parameters, and accuracy in wikis, paper notebooks, or shared spreadsheets. That makes it hard to reproduce a model or trace it back to its data, creator, and training conditions.
Marsden proposes four requirements for an MLOps pipeline. Training and deployment must be reproducible, so another person should be able to retrain an old model months later with sufficiently equivalent data, code, and hardware. The process must be accountable, with a record of the data, person, and other factors behind a model. It must support asynchronous collaboration, allowing colleagues to fork work and make changes without interfering with one another. Finally, model development and deployment must be continuous, with automatic deployment and statistical monitoring rather than manual movement of notebooks or serialized model files.
Model monitoring needs data about predictions, not only service health
Marsden explains that model monitoring differs from monitoring a normal microservice. Latency and error rates can look normal while a model's predictions become wrong. Production data is often unlabeled, since having the correct answer in advance would remove the need for the model. He gives the example of a road-sign classifier for autonomous vehicles. A model trained without stop signs in snow might fail when snow covers a large region. One possible response is to monitor the statistical distribution of classifications and alert a person when it drifts from the expected training distribution. The team can then investigate, roll back, or retrain the model.
Machine learning has more versioned inputs than ordinary software
The software lifecycle can often reproduce a deployment from a code version, dependencies, and a container definition. Machine learning adds data, data-processing code, training parameters, and intermediate datasets. Marsden calls each execution a run, whether or not the team names it. A run records the code and environment used, its input data and parameters, and its outputs. Data runs can represent feature engineering or data splitting. Model runs produce serialized models and metrics from validation data. Those model artifacts can then be deployed and monitored, with monitoring or new data sending the process back through different parts of the lifecycle.
Distributed data science needs GitHub-style collaboration around runs
Marsden separates collaboration into synchronous work, where people share a room or environment, and asynchronous work, where each person works on a separate copy. He uses GitHub's familiar pattern as the model: fork or branch a project, make changes independently, update from the main version, resolve conflicts, and propose the change for review. Applying this to machine learning requires more than code versioning. The system also needs to track datasets, models, metrics, parameters, and development environments. Dotscience packages these items into runs so colleagues can compare work and review changes without a live call, including across time zones.
Existing tools cover separate parts of the MLOps problem
Marsden says teams should find tools and use them rather than waiting for one perfect system. He names Dotscience for the product being demonstrated, MLflow for experiment tracking, Weights & Biases for comparing metrics and hyperparameters, DVC and Pachyderm for data version control, and Mbed for merging notebooks and data. His demo shows runs with different optimizers, epochs, and accuracy scores, a provenance history for a road-sign dataset, input and output files, and a model artifact ready for deployment. The attempted update and merge operation does not finish during the session.
Infrastructure provisioning belongs inside the MLOps platform
In response to a question, Marsden places repeatable infrastructure and platform provisioning within an MLOps platform. He describes a Terraform repository for deploying Dotscience, after which data scientists can self-serve compute from the underlying cloud provider. They can request a GPU for training, switch back to CPU resources, and have idle machines shut down after a period to control costs. He also says the platform should attach compute from multiple clouds, on-premises systems, or a machine on a user's desk. He connects this infrastructure flexibility to storage and network virtualization.
"MLOps is not just about operating models it's actually about the entire lifecycle of doing data engineering training models and then getting models into production."Luke Marsden31:02
Who should watch
You can train models locally, but deployment takes months or fails when the model reaches the production boundary.
Your data science team shares notebooks, folders, spreadsheets, or environments and needs a way to work asynchronously.
You are deciding what an MLOps platform should cover, including experiment tracking, provenance, monitoring, and flexible compute.
