# ML Observability

Aparna Dhinakaran, Arize AI | MLOps Meetup | Episode 27 | 55:04
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=joTF9BRwWp4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/ml-observability
Published: 2020-07-26
Tags: data-quality, drift, monitoring, observability

## TL;DR
- Machine learning monitoring has to cover changing input data and model outputs, since models depend on conditions that can shift after deployment.
- Teams should connect delayed actual outcomes back to predictions so they can measure performance and investigate failures in production.
- Aparna Dhinakaran uses sliced metrics, prediction-versus-actual comparisons, and reference distributions to trace a California fraud-model issue to a feature value absent from training data.

## Summary
Aparna Dhinakaran explains why deployed machine learning models need different monitoring from ordinary software. Model behavior depends on incoming data, which can change in ways that code monitoring will not catch. She describes common failures such as drift, new feature values, distribution changes, data quality problems, bias, and performance differences across user groups. Her Arize demo follows a fraud model whose accuracy falls for California loans. By filtering metrics, comparing predictions with actual outcomes, and checking production distributions against training data, she finds a new feature value that the model had not seen during training. Aparna also discusses the practical work around teams use today, including scripts and dashboards, and the uncertainty over who owns a model after deployment. She recommends capturing inputs, predictions, and actual outcomes, then tuning notifications to the model and business context. The talk also covers unusual conditions during COVID-19, when historical patterns stopped being reliable.

## Key ideas
### Model monitoring must cover data as well as code
[09:03](https://www.youtube.com/watch?v=joTF9BRwWp4&t=543s)
Aparna says software monitoring and model monitoring are different because a model depends on incoming data. New inputs or changing inputs can push the model into situations it did not encounter during training. That creates more things to inspect than a linear software system. She describes model monitoring as a way to understand whether decisions remain reliable for particular states, user groups, device versions, or other slices of production traffic. A single global metric such as accuracy cannot show whether one important population is experiencing a serious failure.

### Deployment is the start of operational work
[13:13](https://www.youtube.com/watch?v=joTF9BRwWp4&t=793s)
Teams often treat production deployment as the end of the model lifecycle, then discover they cannot answer whether the model is working. Aparna describes leaders who have spent millions on machine learning but still do not know if their models perform in production. The problem includes validating a research model before deployment, monitoring it after release, finding drift, identifying distribution changes, and investigating data quality issues. She compares this with software development, where testing, integration, and production monitoring are more established.

### Model ownership is an organisational decision
[16:35](https://www.youtube.com/watch?v=joTF9BRwWp4&t=995s)
The person responsible for a production model depends on how a company works. Some teams use a 'you build it, you own it' model, where the data scientist keeps responsibility after deployment. Other teams have data scientists who are not production engineers, so machine learning engineers combine machine learning and software engineering skills to deploy and operate models. Aparna says teams commonly rely on dashboards, Python scripts, or manual data pulls when an issue appears. Those workarounds leave unclear ownership and consume time that could go into improving models.

### Actual outcomes complete the monitoring loop
[21:04](https://www.youtube.com/watch?v=joTF9BRwWp4&t=1264s)
Aparna recommends capturing model inputs, predictions, and actual outcomes. The actual outcome may arrive much later, such as a fraud result returned 30, 60, or 90 days after a prediction. It may also live in a separate database from the model data. Connecting the outcome back to the original prediction makes it possible to calculate performance, track changes, and alert on real failures. When actuals are unavailable, she suggests looking for proxy metrics that can provide some signal about model behaviour.

### Sliced analysis can turn a vague ticket into a diagnosis
[24:03](https://www.youtube.com/watch?v=joTF9BRwWp4&t=1443s)
In the Arize demo, a team receives reports that a loan fraud model is failing in California. Aparna filters the accuracy view to that state and finds a dip between the 14th and 15th. She then compares global predictions with California predictions and compares both against actual outcomes. The slice shows that the model predicts more fraud in California than actually occurs, which narrows the problem to false positives. She uses the model's important features to continue investigating instead of stopping at the global accuracy number.

### Reference distributions expose inputs the model never saw
[31:07](https://www.youtube.com/watch?v=joTF9BRwWp4&t=1867s)
Aparna compares production feature distributions with the training or validation distribution. For the California slice, the production data contains a 'tax payment' value that was absent from the training data. She then checks the accuracy for that specific slice and finds it is much lower than the global result. This gives the team a concrete explanation for the earlier performance drop. The example shows how distribution checks and slice-level metrics can reveal a data change behind a model failure.

### Extreme events can invalidate historical expectations
[35:10](https://www.youtube.com/watch?v=joTF9BRwWp4&t=2110s)
Aparna connects model monitoring to the COVID-19 period. Inputs changed sharply, unemployment produced events far outside normal historical patterns, and traffic fell to a small fraction of its earlier volume. Teams told her that weather, banking, and other predictions were no longer behaving as expected. Her monitoring advice covers input-level checks, model-response checks, and conditions that fall outside the environment represented in the training data. A model that looked acceptable under historical conditions may need closer inspection when the surrounding world changes.

### Notifications need model-specific thresholds
[39:04](https://www.youtube.com/watch?v=joTF9BRwWp4&t=2344s)
Aparna warns that poorly configured alerts can page people constantly. The acceptable performance range depends on the model and its business use. One team may tolerate a five or ten percent decline over a week, while another may need to react to a smaller change. Notifications should cover evaluation metrics and unusual inputs, such as a large number of feature values never seen before. She also distinguishes between being notified that something deserves investigation and being paged as if the system is immediately failing.

## Notable quotes
- Aparna Dhinakaran: "Software monitoring is just not like model monitoring at all." (09:03)
- Aparna Dhinakaran: "What you really should do is connect those actuals back to the predictions so that you can build these metrics and you can have these kind of performance tracking and alerts." (21:36)
- Aparna Dhinakaran: "When I actually went and trained my model, I didn't really have any kind of values for tax payment as one of the feature values for this feature, but I'm seeing it in production." (32:55)
- Aparna Dhinakaran: "You have a model, you're going to put it into production, it matters to you, it matters to the business, so you should monitor it." (49:47)

## Tools & references mentioned
- Arize AI
- Arize
- Uber
- Michelangelo
- Manifold
- Y Combinator
- MonitorML
- Grafana
- Prometheus
- Lending Club
- Kaggle
- COVID-19
- Cornell University
- University of California, Berkeley
- SR11
- SHAP values
- LIME values

## Who should watch
- You have models in production and people discover failures through tickets, downstream systems, or customer complaints.
- Your team has dashboards or scripts but cannot connect production performance to the data and outcomes that caused it.
- You need a practical way to investigate drift, new feature values, or poor performance in a specific slice of users.

## Editor's note

From the pack [Monitoring and drift](https://mlopstalks.com/packs/monitoring-and-drift):

Gar needs the training-data version that belongs to the deployed model, and Paka describes how missing model records made comparisons hard to trust. ZenML records pipeline steps, inputs, outputs and code versions for each run, giving investigations a record of how a model was produced. Live request capture, outcome joins and alert thresholds still need to be designed for the service being monitored.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Model Watching: Keeping Your Project in Production](https://mlopstalks.com/talks/model-watching-keeping-your-project-in-production) (Ben Wilson, Databricks, 53:08)
- [Model Performance Monitoring and Why You Need it Yesterday](https://mlopstalks.com/talks/model-performance-monitoring-and-why-you-need-it-yesterday) (Amit Paka, Fiddler AI, 1:06:51)
- [The Not So Talked About Reasons Model Monitoring Fails](https://mlopstalks.com/talks/the-not-so-talked-about-reasons-model-monitoring-fails) (Oren Razon, Superwise, 56:03)
- [Model Monitoring in Practice: Top Trends](https://mlopstalks.com/talks/model-monitoring-in-practice-top-trends) (Krishnaram Kenthapadi, Fiddler AI, 51:34)
- [Model Monitoring: The Million Dollar Problem](https://mlopstalks.com/talks/model-monitoring-the-million-dollar-problem) (, 52:55)