# Continuous Integration for ML

Elle O'Brien, Iterative | MLOps Coffee Sessions | Episode 6 | 1:01:46
Hosted by David Aponte

Source: https://www.youtube.com/watch?v=L98VxJDHXMM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/continuous-integration-for-ml
Published: 2020-08-10
Tags: evals, testing

## TL;DR
- Elle O'Brien says academic research often lacks code review, shared repositories, and frequent checks, while industry software teams may deploy several times a day.
- Continuous integration for ML must account for data versioning, model compute needs, infrastructure, and useful evaluation reports alongside source code.
- CML builds on GitHub Actions and GitLab CI to connect data versioning, cloud compute, and human-readable reports to an ML workflow.

## Summary
Elle O'Brien describes her move from academic research in computational neuroscience to data science at Iterative. She found a large gap between research coding, which may never receive review, and software development, where teams check in, test, merge, and deploy work frequently. David Aponte and Elle then explain continuous integration and continuous delivery in practical terms. ML adds complications because models depend on data, training can require substantial compute, infrastructure affects reproducibility, and a simple pass or fail signal often does not explain model quality. Elle explains how DVC extends Git workflows to large datasets and how CML uses existing CI systems rather than replacing them. CML can provision cloud machines for training and place plots, tables, and other artifacts in merge request reports. Elle closes with a systems-focused lesson: failures should lead to blameless post-mortems that ask whether the process, automation, or team structure made the mistake likely.

## Key ideas
### Academic research and production software reward different working habits
[04:43](https://www.youtube.com/watch?v=L98VxJDHXMM&t=283s)
Elle O'Brien says the transition from academic research to industry was much bigger than she expected. As a student and scientist, she wrote code, but nobody usually checked it, reviewed it, or tried to reproduce it. At Iterative, people may deploy software several times a day. The workflow includes checking code in, creating branches, discussing changes, merging them, and testing continuously. She misses the freedom to spend a month exploring a research tangent, while also appreciating the feedback and collaboration that come from an organized engineering process.

### A statistics background helps Elle judge whether a model is plausible
[06:30](https://www.youtube.com/watch?v=L98VxJDHXMM&t=390s)
Elle says her years in research gave her a strong statistical foundation. She describes one benefit as being able to look at a model and think, "this is probably junk," based on experience rather than only following a procedure. She sees research training and production engineering as complementary, although labs often do not reward production-quality code. Her current work focuses more on tools, so she sometimes misses making scientific inferences. David agrees that understanding both the model and how to put it into production is valuable.

### Continuous integration means checking in and testing work frequently
[14:07](https://www.youtube.com/watch?v=L98VxJDHXMM&t=847s)
David explains continuous integration as the practice of having several people make changes on branches and regularly integrate those changes into a main codebase. The aim is to avoid long-lived branches, large merge conflicts, and surprises after integration. Elle adds that each check-in should also test the code. The DevOps habit is to keep practicing what it would be like to have the change in production, instead of waiting until the end of a project to discover problems.

### Continuous delivery makes production feedback available sooner
[17:35](https://www.youtube.com/watch?v=L98VxJDHXMM&t=1055s)
David defines continuous delivery as keeping the ability to release available at any time and increasing the speed and frequency of releases. Frequent releases can reduce the risk and cost of waiting while problems accumulate. Elle gives an example from a startup where she was the only data scientist. She could create a new model, but deployment depended on another person's queue of work, so the model eventually went stale without being tested in production. Continuous delivery gives data scientists faster feedback about dependencies, behavior, and business value.

### Git workflows need extensions for ML data and infrastructure
[31:20](https://www.youtube.com/watch?v=L98VxJDHXMM&t=1880s)
Elle explains that GitHub, GitLab, and Bitbucket evolved around complex, changing software projects with branches, dependencies, and multiple contributors. ML projects add data and infrastructure dependencies that are harder to track. DVC extends Git versioning to large datasets while keeping the data in storage such as an S3 bucket and storing a link in the repository. With data under source control, dataset changes can trigger CI. Without that link, CI only sees source-code changes.

### ML training can exceed the resources available in ordinary CI runners
[34:23](https://www.youtube.com/watch?v=L98VxJDHXMM&t=2063s)
ML models may require much more memory or compute than a normal GitHub Actions or GitLab CI runner provides. Elle says even scikit-learn work can run out of memory on a laptop, while many applications need GPUs. Teams can use self-hosted runners on premises or in the cloud. CML also includes a use case built with Docker Machine that can start an EC2 instance with the requested GPU, run the training job, and turn the instance off afterward.

### Model evaluation needs more than a binary CI result
[45:14](https://www.youtube.com/watch?v=L98VxJDHXMM&t=2714s)
Elle says standard CI systems often report whether a job passed or failed. That signal is insufficient for many ML decisions. Teams may need detailed metrics, visualizations, and performance checks on important subsets of the data. They also need to inspect which features are being used and whether that is acceptable. CML reports can include PNG files, tables, and other artifacts in a pull or merge request. The approach does not depend on a particular visualization library or on whether the code uses Python or R.

### Blameless post-mortems should examine the system around a failure
[59:41](https://www.youtube.com/watch?v=L98VxJDHXMM&t=3581s)
Elle's main DevOps lesson is to investigate failures without placing all responsibility on the person who made the final mistake. A missed step may have been better automated. A manual process may have imposed too much cognitive load. A handoff may have been delayed by a bottleneck or by competing team priorities. She recommends asking whether people were set up to move quickly and whether the system could have prevented the failure. David connects this to the idea that a person is not the same thing as their code.

## Notable quotes
- Elle O'Brien: "I just feel like the standards are different." (05:00)
- Elle O'Brien: "You want to be checking in pretty frequently." (15:08)
- Elle O'Brien: "DVC is basically a tool for extending Git versioning to large data sets." (32:00)
- Elle O'Brien: "ML is really challenging to, you often need in addition to some really detailed metrics being reported, data visualization helps a lot." (45:33)
- Elle O'Brien: "When people are failing in your project, try to take the most blameless post-mortem possible." (59:41)

## Tools & references mentioned
- Iterative
- DVC
- Git
- GitHub
- GitLab
- Bitbucket
- GitHub Actions
- GitLab CI
- CML
- Docker Machine
- S3
- EC2
- Terraform
- The DevOps Handbook
- Botnik Studios
- Git LFS

## Who should watch
- You are a data scientist whose models often wait on another team before anyone tests them in a production-like environment.
- Your ML repository contains large datasets or expensive training jobs, and ordinary Git-based CI does not handle the data or compute well.
- You want a practical introduction to CI/CD without adopting a separate ML platform, and you are considering DVC or CML.

## Editor's note

Elle O'Brien says ML projects add data and infrastructure dependencies that are harder to track than ordinary software dependencies. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a model can be traced to the data and code that produced it. Its stack configuration also lets the same pipeline run across different infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [CI/CD & Continuous Training in ML](https://mlopstalks.com/talks/ci-cd-continuous-training-in-ml) (David Hershey, Determined AI, 1:00:53)
- [Continuous Delivery and Automation Pipelines in Machine Learning, Part 1](https://mlopstalks.com/talks/continuous-delivery-and-automation-pipelines-in-machine-learning-part-1) (, 59:05)
- [Continuous Delivery and Automation Pipelines in ML, Part 2](https://mlopstalks.com/talks/continuous-delivery-and-automation-pipelines-in-ml-part-2) (, 1:07:48)
- [A Missing Link in the ML Infrastructure Stack](https://mlopstalks.com/talks/a-missing-link-in-the-ml-infrastructure-stack) (Josh Tobin, Stealth-Stage Startup, 56:03)
- [DevOps, Security, and Observability in ML](https://mlopstalks.com/talks/devops-security-and-observability-in-ml) (Luke Marsden, MLOps Consulting, 32:46)