# Engineering MLOps

Emmanuel Raj, TietoEvry | MLOps Meetup | Episode 69 | 51:55
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=UhoEJxG0duc
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/engineering-mlops
Published: 2021-06-28
Tags: governance, human-in-the-loop, monitoring, testing

## TL;DR
- Machine learning projects often fail because teams lack a systematic way to develop AI software, plan for continual learning, govern their operations, and test real-world behavior.
- A production MLOps workflow should treat the pipeline as the product, combine automated checks with human oversight, and use a release strategy that fits the business problem.
- MLOps build and deployment practices have matured faster than monitoring, where teams still need better business metrics, error handling, security, and predictive capabilities from AIOps.

## Summary
Emmanuel Raj explains MLOps as the use of software engineering and DevOps practices to build, deploy, and monitor machine learning systems. He connects project failures to weak development processes, a lack of retraining after deployment, unsuitable model release strategies, limited governance, and testing that stops at unit and integration tests. Raj recommends treating the pipeline as the product, with quality checks, load testing, human review, and release methods such as A/B, shadow, or canary testing. Teams adopting MLOps should start from their existing DevOps stack instead of buying tools without a clear need. He also describes AIOps applications, including predicting whether a release will succeed, detecting anomalies in logs, and triggering error-handling scripts. The practical demonstration uses a FastAPI weather prediction service and Locust to load test it before production. Raj considers model building and deployment more mature than monitoring.

## Key ideas
### MLOps addresses the process failures around machine learning software
[06:38](https://www.youtube.com/watch?v=UhoEJxG0duc&t=398s)
Raj says teams often begin with data scientists building models in notebooks and deploying them in a basic way to a cluster or other target. The deeper problem is that companies lack a systematic way to develop AI software. They may prove business value with a quick proof of concept, then fail to plan for continual learning and model retraining in production. Other failures come from choosing a release strategy that does not fit the business KPIs, weak governance for data and models, poor lineage and auditing, and testing that does not cover real-world conditions such as load. Framework differences also made model serialization and packaging difficult before tools such as Open Neural Network Exchange became available.

### Notebooks are useful for development, while production pipelines should use scripts
[10:49](https://www.youtube.com/watch?v=UhoEJxG0duc&t=649s)
Raj says Jupyter notebooks provide a web interface for viewing outputs during development, but that interface is unnecessary inside a CI/CD pipeline. Software delivery pipelines usually work with Python scripts and other files that can be deployed and run directly. He recommends keeping this part of the workflow lean and simple. He also says Open Neural Network Exchange is not a complete solution for model interoperability. It supports common frameworks such as scikit-learn and TensorFlow, while custom models may still be impossible to serialize with it. Raj describes it as a useful partial solution rather than a universal answer.

### The pipeline should be treated as the product
[12:35](https://www.youtube.com/watch?v=UhoEJxG0duc&t=755s)
Raj describes robust CI/CD as a change in mindset from building a model to building a pipeline. Each phase needs quality assurance, including development, QA, and deployment. QA should include load, stress, and integration testing before release, while some checks should retain human oversight because fully automating every step can allow failures into production. The release method should match the business problem, with options such as A/B testing, shadow testing, or canary testing. Teams should automate as much as is appropriate, involve people where quality assurance requires judgment, and monitor business outcomes as well as technical behavior.

### AIOps can predict failures and automate parts of incident handling
[15:56](https://www.youtube.com/watch?v=UhoEJxG0duc&t=956s)
Raj distinguishes MLOps, which applies DevOps practices to machine learning deployment and monitoring, from AIOps, which applies AI to DevOps practices. One possible use is a predictive model trained on historical release data to estimate whether a new model release is likely to succeed. Another is anomaly detection over historical service logs, so the system can identify patterns that may precede an error or attack. If an anomaly is detected, a script in the CI/CD workflow could rerun a service, redeploy another model, or take another predefined action. The quality of these alerts depends on documenting and storing useful historical data.

### Teams should build from their existing stack instead of choosing tools first
[19:39](https://www.youtube.com/watch?v=UhoEJxG0duc&t=1179s)
For a company with models already in production, Raj recommends first understanding its existing tools and workflow. The first phase is an ML platform with lineage for data, source code, and models. The second is deployment, where trained and serialized models are packaged as microservices or another suitable format. Monitoring follows as a separate phase. Raj advises teams to build on their existing DevOps services when they fit, keep vendor dependence and cost in mind, and consider open-source options such as MLflow. His advice is to focus on the problem and the most effective workflow rather than falling in love with a particular tool.

### Monitoring must cover both model behavior and application behavior
[27:20](https://www.youtube.com/watch?v=UhoEJxG0duc&t=1640s)
Raj divides monitoring into model performance and application performance. Tools such as Prometheus and Grafana can provide an overview, but the engineer still has to decide which metrics matter for the business problem. He names response time, failure rate, logs, and other application performance measures as examples. Monitoring should also support decisions about retraining and model release. Raj says teams can add anomaly detection and AIOps when they want to predict failures or automate responses. He recommends starting with the business value and selecting metrics from there, rather than assuming a monitoring tool determines what should be measured.

### Build and deployment are further along than monitoring
[32:45](https://www.youtube.com/watch?v=UhoEJxG0duc&t=1965s)
Raj describes the high-level MLOps workflow as building models, deploying models, and monitoring models. The build stage has developed governance practices that can trace the data, source code, and parameters used to train a model. Deployment has also benefited from technologies such as Kubernetes and Docker, along with several available deployment targets. Monitoring still needs more work. Teams need to customize inference metrics for each business problem, choose suitable release strategies, and plan how to handle errors. Raj expects progress from combining MLOps with AIOps, especially for predictive release checks, log monitoring, automatic error handling, and security.

### Load testing belongs in quality assurance before a model reaches production
[34:42](https://www.youtube.com/watch?v=UhoEJxG0duc&t=2082s)
Raj's demonstration uses a FastAPI microservice that predicts whether it will rain from inputs including temperature, humidity, wind speed, visibility, pressure, and current weather conditions. The service loads a support vector machine and a scaler, then returns a prediction through a REST endpoint. He uses Locust to send repeated requests and examine response times, request rates, active users, and failures. In the example, the test simulates 50 users with 10 users added per second. Raj says the resulting statistics can be logged by a script in a QA or development stage. A clean result can allow automatic deployment, while failures should stop the process for human inspection.

## Notable quotes
- Emmanuel Raj: "It's a mindset shift from building a model focusing on building a model to building a pipeline and not seeing model as your final product but seeing pipeline as your final product." (12:35)
- Emmanuel Raj: "Don't fall in love with tools, fall in love with the most optimal way to solve a problem." (24:44)
- Emmanuel Raj: "There are two things to monitor especially which is the model performance and the application performance on the high level." (27:50)
- Emmanuel Raj: "One important test that we should do is a stress test or load testing before you push out your model to production." (34:42)

## Tools & references mentioned
- Engineering MLOps
- Open Neural Network Exchange
- Jupyter notebooks
- AIOps
- MLflow
- FastAPI
- Docker
- Kubernetes
- Prometheus
- Grafana
- Evidently AI
- Locust
- GitHub
- GitLab
- Azure
- Amazon

## Who should watch
- You have machine learning models in production but lack a clear build, release, monitoring, or retraining process.
- Your team is choosing MLOps tools and needs a way to start from its existing DevOps stack and business constraints.
- You want a practical example of load testing a machine learning microservice before deployment.

## Editor's note

Emmanuel Raj says poor lineage and auditing can leave teams unable to trace the data, source code, and models behind a production system. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a model or artifact can be traced back to what produced it. That record also supports repeatable pipeline work across infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLOps Insights](https://mlopstalks.com/talks/mlops-insights) (David Aponte-Demetrios Brinkmann-Vishnu Rachakonda, 37:47)
- [Practical MLOps Part 2](https://mlopstalks.com/talks/practical-mlops-part-2) (Alfredo Deza, Author and Speaker, 1:01:38)
- [Scaling AI in Production](https://mlopstalks.com/talks/scaling-ai-in-production) (Srivatsan Srinivasan, AIEngineering, 51:56)
- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)
- [Operationalize Machine Learning at Scale with MLOps](https://mlopstalks.com/talks/operationalize-machine-learning-at-scale-with-mlops) (Christopher Bergh, DataKitchen, 57:50)