# MLOps - The Blind Men and the Elephant

Saurav Chakravorty, Brillo | MLOps Meetup | Episode 10 | 55:02
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=RTBq7e3FhEw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mlops-the-blind-men-and-the-elephant
Published: 2020-05-11
Tags: build-vs-buy, drift, monitoring, orchestration, platform-teams

## TL;DR
- Different groups define MLOps from their own needs, so a platform must support business analysts, software engineers, machine learning engineers, and data scientists.
- A financial services team reduced the time from an idea to an experiment and production deployment from about six months to about three weeks by improving its data and ML delivery processes.
- Managed ML services remove much of the operational work, while teams that build their own systems must handle patching, security, backups, firewalls, and audits.

## Summary
Saurav Chakravorty explains MLOps through the story of blind men describing an elephant. Business analysts want fast answers and may accept lower reproducibility. Software engineers prefer production-quality code but may spend less time exploring data. Machine learning engineers combine software and ML skills, while data scientists often feel the greatest pain when moving models from notebooks into production. He describes a financial services team where feature engineering, environment changes, recoding, and testing made the path from idea to production take about six months. A feature store, Spark ETL, Airflow, and a more consistent delivery process reduced that time to about three weeks. Saurav also discusses the limits of building an internal platform. A working MLflow server is easy to install, but production use brings questions about security, patching, backups, firewalls, and audits. He recommends managed services for teams that do not want to own that operational burden. He also argues that production models need continuous data checks for drift, even when they are not retrained continuously.

## Key ideas
### MLOps looks different to each group using an ML platform
[01:57](https://www.youtube.com/watch?v=RTBq7e3FhEw&t=117s)
Saurav compares ML practitioners to the blind men in the elephant story because each customer sees MLOps from a different part of the problem. Business analysts understand their business area and use forecasting, regression, and clustering, but they often value speed over reproducibility. Software engineers prefer production-quality integration and deployment code, although they may spend less time exploring data. Machine learning engineers already combine software and ML practices. Data scientists often sit between these groups and feel the strongest pain when moving work from a notebook into production.

### Data scientists lose time when model work must be rewritten for production
[07:09](https://www.youtube.com/watch?v=RTBq7e3FhEw&t=429s)
The data scientists Saurav describes developed models in R or Python and then worked with a development team to convert them into Java or SQL. Model parameters could end up embedded in SQL code. This process took months and left the scientists spending much of their time helping a model move through the production pipeline. MLOps changed that division of work by removing much of the post-model-building conversion effort. The data scientists could spend more time running experiments instead of translating their work for another environment.

### A financial services workflow took about six months from idea to production
[09:41](https://www.youtube.com/watch?v=RTBq7e3FhEw&t=581s)
One financial services customer needed about six months to add a feature and put the resulting model change into production. Feature engineering took two or three months. Moving the new data through the environments took about another month, followed by recoding and testing. Testing itself took time because the business impact of changes could take a long period to appear. The process was slow even though the team had strong engineering skills and already had its own technology stack.

### ML needs representative data during development
[11:37](https://www.youtube.com/watch?v=RTBq7e3FhEw&t=697s)
The customer could not make all production data available in the development environment because of security and compliance constraints. Saurav says this creates a difference between ordinary software delivery and ML delivery. Code can pass through development and pre-production, but a model also needs feedback about accuracy and related metrics. The team separated the training run from the code build process. Users checked in training code, triggered a pipeline, and received a report after the model build completed. A separate environment allowed approved users to work with more realistic data under the relevant rules.

### A platform that works technically still needs operational ownership
[20:09](https://www.youtube.com/watch?v=RTBq7e3FhEw&t=1209s)
Saurav says it is easy to assemble an open-source system from documented components and get it working. That does not answer who patches the virtual machines, handles backups, configures the firewall, or responds to an audit. He gives the example of hosting an MLflow server on a virtual machine and receiving difficult questions from information security teams about its maintenance and protection. Data scientists and ML engineers may not have the network security and infrastructure experience needed to own those tasks. Managed services are easier for teams that do not want to take on that responsibility.

### The six-month delivery time fell to about three weeks
[25:48](https://www.youtube.com/watch?v=RTBq7e3FhEw&t=1548s)
The financial services team reduced the idea-to-production process from about six months to about three weeks. Saurav attributes the change to applying delivery practices across the Spark data pipelines and the machine learning code. The original delay included months spent building data pipes, creating database snapshots, running experiments, and testing the result. The new process made experimentation easier by giving data and ML code a more consistent path through the system. The reduction mattered because it allowed the team to try more experiments.

### Production models need ongoing checks even when they are not retrained continuously
[29:49](https://www.youtube.com/watch?v=RTBq7e3FhEw&t=1789s)
Saurav says the team does not continuously retrain every production model. It does need to inspect incoming data continuously. He describes comparing each scored batch with a historical or business-defined profile, then checking whether the data still resembles the data used to build the model. Changes in feature values, accuracy, or fairness can indicate that the model's assumptions no longer hold. He connects this to domain drift, where the underlying data distribution changes over time. The monitoring system should help decide when retraining is needed instead of treating every change as an automatic retraining trigger.

### A strong image classifier can still fail in the field
[40:07](https://www.youtube.com/watch?v=RTBq7e3FhEw&t=2407s)
Saurav describes a grain-classification project intended to estimate the quality and price of a large shipment from an image of a small sample. The classifier performed well in testing, but the field system did not replace the expert's judgment. The person assessing grain normally holds it in their hand and uses more than visual information. When that person judged an image instead, the answer differed from the judgment made using the physical sample. The model was effective at classifying images, yet the complete system failed to deliver the measure of quality needed in practice. Saurav uses this example to show why the operating context matters.

## Notable quotes
- Saurav Chakravorty: "Each of them looked at MLOps in a different way." (08:53)
- Saurav Chakravorty: "I can just install a server, connect to this, spend some money, create some APIs, but what if it gets audited?" (22:03)
- Saurav Chakravorty: "We may not train it continuously, but we should be looking at that data set continuously." (30:22)
- Saurav Chakravorty: "It is easy to hack something together, but once you need to really get it out there on the production level, it's a whole different story." (22:23)
- Saurav Chakravorty: "Even though our classifier does very well on an image, it could never replace the accuracy which people got by taking it into their hands." (41:39)

## Tools & references mentioned
- MLOps Community
- MLflow
- Azure
- Databricks
- Spark
- Airflow
- Kubeflow
- AWS
- GDPR
- COVID-19

## Who should watch
- You are a data scientist whose models spend months being rewritten or moved between development and production environments.
- Your team is choosing between assembling open-source MLOps components and using a managed service, and you need to understand the operational work behind each option.
- You are designing monitoring for a production model and need to detect changes in incoming data without automatically retraining after every change.

## Editor's note

Saurav Chakravorty describes data scientists losing months when notebook models in R or Python have to be recoded in Java or SQL for production. ZenML lets teams write ML workflows as Python steps and run the same pipeline code across different infrastructure. Each run records its inputs, outputs, and code version, so the resulting model can be traced to the work that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLOps: Isn't That Just DevOps?](https://mlopstalks.com/talks/mlops-isnt-that-just-devops) (Ryan Dawson, Seldon, 1:06:32)
- [MLOps Insights](https://mlopstalks.com/talks/mlops-insights) (David Aponte-Demetrios Brinkmann-Vishnu Rachakonda, 37:47)
- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)
- [The Current MLOps Landscape](https://mlopstalks.com/talks/the-current-mlops-landscape) (Nathan Benaich, Air Street Capital & Timothy Chen, Essence VC, 58:31)
- [I Am Once Again Asking "What is MLOps?"](https://mlopstalks.com/talks/i-am-once-again-asking-what-is-mlops) (Oleksandr Stasyk, Synthesia, 1:06:22)