# What MLOps Has Taught Me

Ewan Nicolson, Forecast | MLOps Meetup | Episode 75 | 54:14
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=VMK5jVT9Rk0
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/what-mlops-has-taught-me
Published: 2021-08-20
Tags: data-quality, feature-engineering, feature-stores, recommender-systems, testing

## TL;DR
- MLOps gives teams a shared vocabulary for breaking a machine learning system into parts such as data ingestion, feature stores, models, indexes, and serving.
- Data validation is more useful than fixed integration data alone because production data changes in ways that test fixtures cannot predict.
- Teams work better when people share knowledge across data science, engineering, and architecture instead of handing notebooks or models across rigid silos.

## Summary
Ewan Nicolson uses a local recommender-system demo to explain what production machine learning demands. The system streams Amazon reviews into a data lake, cleans the data, trains a collaborative-filtering model, stores embeddings in an approximate nearest-neighbors index, and serves recommendations through an API. He compares this with how he used to work and discusses data testing, CI/CD, system design, feature stores, notebooks, and team structure. Data tests give more confidence than eyeballing reports, although they are noisy because real data changes. CI/CD for data products remains difficult because training can be expensive and data rollbacks require replaying data and informing downstream users. MLOps also gives teams names for system components, which makes it easier to discuss trade-offs and replace parts. Ewan argues for shared knowledge across roles, while avoiding both rigid silos and the expectation that everyone must master every tool. Collaboration can help teams choose simpler latency and data requirements.

## Key ideas
### A production recommender system has several separate moving parts
[08:04](https://www.youtube.com/watch?v=VMK5jVT9Rk0&t=484s)
Ewan demonstrates a local recommender system built to resemble a real production setup. Streaming data enters from an Amazon reviews dataset and is written to a data lake. Cleaning logic selects the useful columns. A collaborative-filtering model calculates user and item embeddings. Those embeddings go into an approximate nearest-neighbors index, while user and item features go into a feature store. A business rules component queries those stores and exposes recommendations through an API. The demo runs through ingestion, training, index construction, and serving so the audience can see the full path from incoming data to a recommendation.

### Data tests give more confidence than fixed integration fixtures
[13:48](https://www.youtube.com/watch?v=VMK5jVT9Rk0&t=828s)
Ewan says ordinary integration tests are a poor fit for data systems because test data must represent messy, changing production data. Even a carefully prepared fixture misses unexpected cases. Large test runs can also become expensive when they train models or factorize matrices that may exceed available memory. He prefers checks that inspect distributions and results, including Great Expectations tests that express expectations such as values staying within a range with 95 percent confidence. These checks are noisier than standard software tests, so teams still need judgment, but they can catch strange data before it moves further through the system.

### CI/CD for data products is harder because data is expensive to undo
[18:42](https://www.youtube.com/watch?v=VMK5jVT9Rk0&t=1122s)
Ewan thinks CI/CD for data products is close to being practical, but says most of the systems he has worked with are not there yet. Automated tests can be noisy, and model training may require costly cloud resources. Production deployment is also harder to reverse than a software release. If incorrect data reaches a production database, the team may need to replay the data and tell people to replace copies they downloaded to Excel. He says replaying data makes data engineers visibly uncomfortable because it is painful, expensive, and prone to more mistakes. Better tooling could reduce how often teams face this situation.

### MLOps gives teams a vocabulary for decomposing machine learning systems
[21:36](https://www.youtube.com/watch?v=VMK5jVT9Rk0&t=1296s)
Ewan describes older machine learning systems as monoliths that people avoided changing because no one had a shared way to describe their parts. MLOps discussions make it easier to identify separate components for streaming data, data at rest, modeling, feature stores, approximate nearest-neighbors indexes, and APIs. Once a component has a name, teams can discuss whether to build it themselves or use a product such as Pinecone. This makes systems easier to change and lets teams compare alternatives at the right level instead of treating the whole machine learning section as one dangerous box.

### Feature stores are easier to sell through their team benefits than their technical details
[23:23](https://www.youtube.com/watch?v=VMK5jVT9Rk0&t=1403s)
Ewan is skeptical of feature-store pitches that focus on implementation details. A table of technical features did not persuade him because the basic mental model still looked like a familiar key-value store. What matters to him is the shared name and the collaboration it enables. Saying feature store lets data scientists and developers describe the same part of the system without explaining every latency and read-write property. It also signals that one place is intended for shared feature data. Ewan argues that teams should explain how a feature store changes their discussions and working habits, rather than listing its technical capabilities.

### Notebooks can belong in production when their use is controlled
[27:07](https://www.youtube.com/watch?v=VMK5jVT9Rk0&t=1627s)
Ewan accepts that notebooks have real problems, including hidden state and poor behavior in version control, but he does not think they must be banned. He uses one notebook for data-validation visualizations and exploratory analysis, then automates and parameterizes it with Papermill. He uses another while developing the model, changing parameters and examining the embeddings. That notebook starts from shared data-cleaning and manipulation code rather than duplicating the whole pipeline. His condition is responsible use: teams should know when a notebook is exploratory, import from modules, and move reusable code out of notebooks so the same logic does not diverge in different places.

### Teams should share enough knowledge to avoid both silos and tool overload
[30:08](https://www.youtube.com/watch?v=VMK5jVT9Rk0&t=1808s)
Ewan maps different kinds of work across data science, engineering, and cloud or data architecture. He argues against isolating those roles so completely that one group hands work to another. He gives an example where a data engineer deleted a data scientist's notebooks and rebuilt them from scratch. He also rejects the opposite extreme, where every person is expected to master Kubernetes, Airflow, Docker, Dataflow, Dataproc, BigQuery, and many other technologies. Some specialization helps, but too much makes people spend their effort learning tools instead of solving the problem. Shared concepts and higher-level components can lower that burden.

### Collaboration can change the latency and data trade-off of a model
[32:58](https://www.youtube.com/watch?v=VMK5jVT9Rk0&t=1978s)
Ewan uses a quadrant based on latency and the amount of context needed for an inference. A data scientist working alone may choose very low latency and a large amount of historical data because that data is easy to query during training. That choice creates difficult engineering work around caching, freshness, and millisecond response times. With engineers and data scientists working together, the team may accept less history and simpler user information, or compress years of data into a slower-changing user representation such as an embedding. Shared discussion can reveal a practical shortcut before the system becomes unnecessarily hard to operate.

## Notable quotes
- Ewan Nicolson: "I don't think I could give you a very concrete definition of what MLOps is." (05:23)
- Ewan Nicolson: "Real world data is messy and it changes and it messes you up and it breaks things in weird ways that you could never have anticipated." (14:35)
- Ewan Nicolson: "Data is always just a little bit horrible and a little bit messy." (20:28)
- Ewan Nicolson: "Over specialization is for insects." (40:26)
- Ewan Nicolson: "If you don't have that kind of collaboration, the hive mind, if you don't have people working together like that, then you don't solve the most difficult problems." (42:54)

## Tools & references mentioned
- MLOps Community
- Great Expectations
- Pinecone
- GitLab
- Papermill
- Jupyter notebooks
- Robert Chang
- GitHub

## Who should watch
- You are building a machine learning system and need a concrete view of how ingestion, training, feature data, nearest-neighbors search, and serving fit together.
- Your data tests pass on fixtures, but production inputs keep producing surprising failures or expensive rollbacks.
- Your team is deciding how much to specialize across data science, engineering, and architecture without forcing everyone to learn every tool.

## Editor's note

Ewan Nicolson says CI/CD for data products is harder because incorrect data is expensive to undo, since teams may have to replay it and ask users to replace downloaded copies. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a data or model artifact can be traced back to what produced it. That record gives teams a concrete place to investigate before replaying data.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLOps Insights](https://mlopstalks.com/talks/mlops-insights) (David Aponte-Demetrios Brinkmann-Vishnu Rachakonda, 37:47)
- [MLOps in Practice: Common Challenges and Lessons Learned](https://mlopstalks.com/talks/mlops-in-practice-common-challenges-and-lessons-learned) (Marouen Hizaoui & Mo Basirati, Machine Learning Reply, 1:03:01)
- [The Godfather Of MLOps](https://mlopstalks.com/talks/the-godfather-of-mlops) (D. Sculley, Google, 51:25)
- [Practical MLOps Part 2](https://mlopstalks.com/talks/practical-mlops-part-2) (Alfredo Deza, Author and Speaker, 1:01:38)
- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)