Damian Brady says DevOps and MLOps share the aim of moving an idea into the hands of users and delivering value, although the work needed to produce and operate a model differs.
2
Machine learning projects involve more experimentation, longer training runs, harder evaluation, and possible data drift, so applying software delivery practices without adjustment can create waste.
3
The model-building part needs specialist tools and processes, while traditional DevOps tools remain useful once a model is ready for staged release, monitoring, and promotion.
Summary
Damian Brady describes MLOps as closely related to DevOps at the level of purpose. Both aim to move a solution from an idea into production and make it useful to people. The differences appear in the work behind a machine learning model. Data scientists experiment more, training can take days, model quality is harder to judge, and production data can change through drift. A successful test result does not prove that a model creates value for real users. Brady recommends building an end-to-end pipeline early, even with a dummy model, then using suitable tools for data versioning, training, validation, and long-running jobs. Once a model becomes a release candidate, familiar DevOps methods such as staged releases, A/B tests, canary deployments, monitoring, and rollback become useful. The conversation also covers team cadence, Git, manual evaluation, shadow deployments, retraining, and the need for software engineers and data scientists to understand each other's work.
DevOps and MLOps share the same purpose, while their delivery work differs
Damian Brady says both DevOps and MLOps aim to move a solution from someone's idea into the hands of users and make sure it provides value. The solution might be code, a database update, or a predictive model. The difference appears in how the model is produced and deployed. He agrees that the concepts are closely related, but says the practical work is quite different. The discussion frames DevOps and MLOps as useful mental models rather than completely separate philosophies. The distinction helps teams account for the added complexity of data, randomness, and model behavior.
Machine learning cannot use every software delivery practice unchanged
Brady warns against assuming that every existing DevOps process applies directly to machine learning. A team might try to use Git flow and start a training run for every code change, but that can be wasteful. Machine learning work has more experimentation, especially at the start, and its cadence is different from traditional application development. In software consulting, Brady often began with a small application, a delivery pipeline, and a production release on the first day. A model project may spend weeks testing approaches before anything is ready to release. A pipeline still helps, but it must fit that slower, exploratory process.
Production is the only place where a model meets real user behavior
Brady says production is always a test environment, because the real test begins when a system meets real users and real conditions. A model can perform well on internal validation data and still fail inside an application. The hosts discuss shadow deployment as a safe way to learn from production traffic. A new, versioned model receives all or some of the same traffic as the existing model, while its predictions are discarded and do not affect users. This tests the model with current inputs and checks operational behavior before promotion. The approach is especially useful when generating predictions is inexpensive.
Model value is harder to evaluate than whether software meets a specification
Brady contrasts a conventional application with a predictive model. A basic application can be tested against requirements, checked for errors, and released even if later bugs need fixing. A predictive model needs to produce a better and genuinely useful result, which is harder to measure. The other speaker notes that model metrics such as accuracy or recall may not map directly to business measures such as click-through rates. In one example, the real outcome is only known after an assay in a lab, which creates a slow and expensive feedback loop. A model can look excellent under one framing and poor under another stakeholder's measure.
Machine learning testing may need manual judgment alongside automation
Brady describes how traditional software teams often automate testing to reduce cycle time. He gives the Azure DevOps example of a system that eventually ran 82,000 unit tests on every pull request and moved from releases every 18 months to major releases every three weeks. The conversation then questions whether the same approach fits model evaluation. Some checks need a person who understands the problem because model quality is not captured by a single metric. Manual work is acceptable when it is documented and reproducible. Automating every step can add cost and overhead, especially when computation is limited.
Different team cadences can work when the model is treated as a separate product
Brady says machine learning specialists may work at a different pace from application developers because model production still requires specialist knowledge and experimentation. One option is to treat the model as a third-party library or service with its own release cadence and version. An application might continue using version one while the team evaluates version two, which could require additional inputs. Another option keeps data scientists within the team but releases their work alongside the application less often, perhaps after several sprints. Brady connects this arrangement to DevOps culture, where teams need to understand and respect each other's roles instead of forcing everyone into the same delivery rhythm.
Git concepts support collaboration and data versioning
Brady says data scientists should learn Git, even though versioning data is harder than versioning source code. He names Pachyderm and DVC as tools that use Git concepts to address data versioning. For him, the value is broader than source control. Distributed version control supports collaboration when people work in different places, and pull request descriptions preserve the reasoning behind a change. The conversation argues that data scientists benefit from software engineering practices such as testing, continuous integration, and continuous delivery, while software engineers working on machine learning systems need enough domain knowledge to understand notebooks, experiments, and model behavior.
Specialist tooling fits training, while DevOps tooling fits model release
Brady divides an MLOps pipeline into a machine learning section and a release section. The first includes data gathering and preparation, feature selection, code, algorithms, training runs, and validation. Data and code combine to produce a model, which must be checked for performance, reproducibility, feature versions, and possible bias. Training may run for days or cost tens of thousands of dollars, so hosted CI systems built for short jobs are not a good fit. After validation produces a release candidate, familiar DevOps tools can manage pre-production checks, A/B tests, canary releases, monitoring, and promotion. Model retraining can happen periodically or after a drift threshold is reached.
"I think from that really high level perspective it's about getting some kind of solution from your head into the hands of people and make sure it's valuable and do that really effectively."05:14
Who should watch
You are a DevOps engineer being asked to apply existing delivery pipelines to machine learning work and need to understand where that approach breaks down.
You are a data scientist or machine learning engineer who wants practical reasons to adopt Git, testing, reproducibility, and release processes.
Your team is struggling with different development cadences, expensive training runs, weak feedback from production, or uncertainty about when to retrain a model.
