# Just Build It! Tips for Making ML Engineering and MLOps Real

Andy McMahon, NatWest Group | MLOps Meetup | Episode 91 | 48:17
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=l1uhE9fEfo8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/just-build-it-tips-for-making-ml-engineering-and-mlops-real
Published: 2022-01-14
Tags: deployment, model-registry, model-serving, monitoring

## TL;DR
- Teams can escape analysis paralysis by choosing a small capability to add to each machine learning project, then building on it in later projects.
- ML engineering gets a model into production, while MLOps manages the process of getting additional models into production and keeping the whole system running.
- A production ML system needs software engineering, deployment automation, model management, monitoring, and clear ownership across data, software, and operations roles.

## Summary
Andy McMahon argues that teams should start building ML engineering and MLOps capability before they feel fully ready. He separates ML engineering, which applies software engineering to get models into production, from MLOps, which manages the wider lifecycle as more models are added. The practical approach is to cross the gap from prototype to production in small steps. A team might begin with version control and packaging, then add unit tests, CI/CD, infrastructure and configuration as code, experiment tracking, model registries, and performance monitoring. Andy also explains what production means, why notebook code needs to be reduced to the minimum needed for data preparation, inference, and post-processing, and how model promotion can use champion and challenger logic. He recommends blended teams that combine data, software, and operations skills, with low-risk projects used to build trust. His advice is deliberately pragmatic: use standards and existing platforms, automate repetitive work, and accept that the system will improve through repeated cycles.

## Key ideas
### ML engineering gets the first model into production, while MLOps handles the next models
[03:45](https://www.youtube.com/watch?v=l1uhE9fEfo8&t=225s)
Andy defines ML engineering as applying enough software engineering to get data science models into production. MLOps takes a wider view of the end-to-end lifecycle and brings together the people building models with operations. He compares the relationship with software engineering and DevOps. The first model is the initial proof that a team can build and run a solution. The next model raises questions about repeatability, promotion, monitoring, and how to manage the process when a model is already live. This distinction gives teams a practical starting point. They do not need to solve every lifecycle problem before putting the first useful model into production.

### Teams should cross the prototype-to-production gap one capability at a time
[09:25](https://www.youtube.com/watch?v=l1uhE9fEfo8&t=565s)
Andy describes a chasm between strong proof-of-concepts and production systems. Teams can end up with models built from data dumps, no way to update them, no reliable hosting method, or no automated training process. His answer is to build the bridge in pieces. A first project might introduce version control and packaging. A later project can add unit tests, followed by CI/CD, infrastructure as code, experiment tracking, and model performance monitoring. The exact sequence depends on what the team already knows. The point is to add a manageable piece, create a foundation, and ratchet up the capability through successive projects.

### Production code should contain the minimum path from data to prediction
[18:04](https://www.youtube.com/watch?v=l1uhE9fEfo8&t=1084s)
Andy says teams should define production before deciding how to reach it. For him, production is an isolated environment with controls, a known risk profile, service-level expectations, and real inference traffic that affects decisions. Notebook code often includes plots, exploratory checks, and tightly connected cells that do not belong in a production service. He recommends choosing a programming approach, separating concerns, and isolating tasks such as data access, feature transformation, model training, inference, and post-processing. Small methods are easier to unit test. In response to Demetrios Brinkmann, he says test-driven development naturally focuses attention on the behavior that matters and leaves exploratory plotting outside the deployed path.

### CI/CD and configuration management remove manual steps from deployment
[25:24](https://www.youtube.com/watch?v=l1uhE9fEfo8&t=1524s)
Andy presents CI/CD as a useful starting point for data scientists who find deployment intimidating. He names GitHub Actions, Jenkins, and AWS pipeline tools as possible choices, while stressing that teams should choose a tool and begin. Automated builds can run unit tests, perform data quality checks, package code, and deploy artifacts. His example pushes an Airflow DAG to S3 after an accepted pull request, with credentials stored as secrets. He also recommends separating configuration from application and modeling code with YAML or JSON. Hydra can manage hierarchical configuration and command-line overrides. This makes environment changes safer because teams can alter configuration without replacing the whole application package.

### Model registries make model promotion explicit and repeatable
[30:34](https://www.youtube.com/watch?v=l1uhE9fEfo8&t=1834s)
For MLOps, Andy starts with model management. He uses MLflow as his example, with experiment tracking and a model registry. Tracking records the experiments and metrics for candidate models. The registry labels models consistently so teams can distinguish development and production versions. That foundation supports model swaps, champion and challenger comparisons, and A/B tests. Andy recommends separating model artifacts from application code, with training and prediction systems treated as separate concerns and the model registry providing the handover. In his example, a new model is compared with the latest production model, and only a successful challenger is promoted to production.

### Monitoring can start with simple drift checks before it becomes an automated system
[32:12](https://www.youtube.com/watch?v=l1uhE9fEfo8&t=1932s)
Andy recommends beginning performance monitoring with a small amount of code that detects drift. He gives Alibi Detect as an example. Writing the first drift check forces the team to define what drift means and what action should follow. Over time, the check can become a scheduled, automated, and orchestrated process. He then describes a model factory made from a training system, model store, production system, and drift detector. That arrangement supports a cycle of training models, checking them, and deciding whether another model should be promoted. Some teams may begin with a train-and-run process that does not persist a model, which can be simpler than starting with full model storage.

### Blended teams can own more of the system without removing specialist support
[34:54](https://www.youtube.com/watch?v=l1uhE9fEfo8&t=2094s)
Andy answers a question from the audience by recommending collaboration with software and DevOps engineers rather than expecting data teams to do everything alone. He describes a useful team mix as people with enough computer science to make systems work, data scientists who can build models, and people who connect the two areas. In a regulated organization, teams should begin with lower-risk use cases. That reduces the cost of mistakes while giving stakeholders evidence that the approach works. He also supports involving site reliability engineers or other infrastructure specialists where needed. The important part is shared ownership and communication instead of handing work over a fence.

### The foundation of MLOps is a running software system with data, model care, and recovery paths
[43:28](https://www.youtube.com/watch?v=l1uhE9fEfo8&t=2608s)
When Demetrios asks about fundamentals beyond tools and stacks, Andy reduces MLOps to the parts that must work together. The software must run successfully and receive data. The model needs regular care because its behavior can move out of tolerance. The team must check whether the model is still working and have a way to fix it when it is not. Andy connects these concerns to his physics background and describes the work as fighting entropy in the system. He says an ML platform often emerges as teams standardize tools, infrastructure, and ways of working. When several teams reuse it, reliability, scalability, and service ownership become additional concerns.

## Notable quotes
- Andy McMahon: "We can let go of the fear of getting started on this journey." (02:00)
- Andy McMahon: "As long as you're thinking more about the processes these are hooking into and the capability within you and your team, your organization, you'll be able to more confidently swap these in and out." (10:42)
- Andy McMahon: "Don't do it all at once. Don't try and get to the nirvana of MLOps in one step." (12:53)
- Andy McMahon: "The big thing for me is always pull it back to what we're trying to do." (43:28)

## Tools & references mentioned
- NatWest Group
- Aggreko
- Machine Learning Engineering with Python
- Pact
- George Patton
- Yoda
- James Clear
- Atomic Habits
- GitHub Actions
- Jenkins
- AWS
- Airflow
- S3
- Hydra
- Facebook
- MLflow
- Alibi Detect
- Terraform
- AWS CloudFormation
- Google Cloud Platform
- Google
- Hidden Technical Debt in Machine Learning Systems

## Who should watch
- You are moving ML prototypes toward production and need a practical sequence for adding engineering practices without stopping delivery.
- Your data team depends on software, DevOps, or site reliability specialists and needs a model for sharing ownership.
- You are choosing MLOps tools and want to focus first on the processes, deployment path, model lifecycle, and monitoring work they need to support.

## Editor's note

Andy McMahon says teams should cross the prototype-to-production gap one capability at a time, starting with manageable additions such as version control and packaging. ZenML lets teams write workflows as Python steps and run the same pipeline on different infrastructure through configuration. Each run records its steps, inputs, outputs, and code version, so later projects can be traced and repeated.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Scaling AI in Production](https://mlopstalks.com/talks/scaling-ai-in-production) (Srivatsan Srinivasan, AIEngineering, 51:56)
- [Practical MLOps Part 2](https://mlopstalks.com/talks/practical-mlops-part-2) (Alfredo Deza, Author and Speaker, 1:01:38)
- [Operationalize Machine Learning at Scale with MLOps](https://mlopstalks.com/talks/operationalize-machine-learning-at-scale-with-mlops) (Christopher Bergh, DataKitchen, 57:50)
- [Tecton Round-table // Get your ML Application Into Production](https://mlopstalks.com/talks/tecton-round-table-get-your-ml-application-into-production) (Kevin Stumpf, Derek Salama, Eddie Esquivel & Isaac Cameron, Tecton, 55:42)
- [Engineering MLOps](https://mlopstalks.com/talks/engineering-mlops) (Emmanuel Raj, TietoEvry, 51:55)