# Model Monitoring: The Million Dollar Problem

 | MLOps Meetup | Episode 87 | 52:55
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=5vGWxwf3jr8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/model-monitoring-the-million-dollar-problem
Published: 2021-11-26
Tags: data-quality, drift, monitoring, observability

## TL;DR
- Model monitoring covers both operational health, such as uptime and latency, and functional behavior, such as data quality, drift, bias, and model performance.
- A monitoring system needs captured inputs and outputs, baselines, thresholds, scheduled checks, dashboards, and alerts, with care around personally identifiable information.
- Amazon SageMaker Model Monitor is a strong fit inside an AWS setup, while Evidently AI offers a simpler way to generate data and model reports across existing pipelines.

## Summary
The Loka Team explains why machine learning models can lose performance after deployment. COVID-19 changed usage patterns, supply chains, and data distributions, so models trained on earlier data no longer matched the world around them. The talk separates monitoring into operational checks, such as uptime and latency, and functional checks, such as data quality, drift, bias, and prediction performance. The presenters then demonstrate Amazon SageMaker Model Monitor, a vendor-specific service with scheduled checks and visualizations; an open-source, cloud-agnostic logging approach based on data profiles; and Evidently AI, a Python library that generates interactive or exportable reports. Their examples use a small customer-churn dataset. The closing advice is practical: choose a tool based on the deployment environment, the complexity of the use case, and how well it fits existing systems. The speakers are honest that tool choice often comes down to those constraints and personal preference.

## Key ideas
### Production data can move away from the data used to train a model
[00:00](https://www.youtube.com/watch?v=5vGWxwf3jr8&t=0s)
The presenters use COVID-19 as a direct example of why production models change over time. The pandemic altered usage patterns, supply-chain throughput, and other conditions, while deployed models had been trained on earlier situations. Production data therefore moved into distributions the models had never seen. They describe this as a reason to monitor changes in distributions and adjust models when needed. Other failures include concept drift, adversarial attempts to exploit a model, outliers, data-quality problems, and failures in upstream or processing pipelines.

### Model monitoring has an operational layer and a functional layer
[01:55](https://www.youtube.com/watch?v=5vGWxwf3jr8&t=115s)
Operational monitoring checks whether the infrastructure is working. The examples include uptime and latency staying within acceptable limits. Functional monitoring examines whether the machine-learning system is behaving as intended. It covers the quality and drift of input data, model configuration and versioning, prediction drift, bias drift, and the model's measured performance. The presenters frame functional monitoring as the part that usually needs closer involvement from data scientists, while operations teams already have infrastructure health to manage.

### A monitoring setup needs captured data, baselines, thresholds, and alerts
[02:40](https://www.youtube.com/watch?v=5vGWxwf3jr8&t=160s)
The reference workflow begins with serving logs, a dataset or feature repository, model artifacts, and metadata. Teams capture model inputs and outputs, while checking whether that creates personally identifiable information concerns. They define baseline statistical distributions, acceptable missing-data thresholds, and a baseline model for comparison. Monitoring can then run on data, the model, or both. Dashboards show how the model and its metrics change over time, and alerts notify the team when a threshold is crossed. The presenters demonstrate these ideas with a customer-churn example based on a small US mobile-provider dataset containing 5,000 records and 20 customer attributes.

### SageMaker Model Monitor provides scheduled checks inside AWS
[05:33](https://www.youtube.com/watch?v=5vGWxwf3jr8&t=333s)
Buen presents Amazon SageMaker Model Monitor as a managed service for continuous monitoring of production models. Its features cover data quality, model quality, model bias through Amazon SageMaker Clarify, and model explainability. A workflow captures a configurable fraction of endpoint inputs and predictions in Amazon S3, creates a baseline job, and attaches monitoring to an endpoint through a schedule. Model quality checks can use ground-truth labels collected from the application. The demo deploys a model, creates a baseline from validation data, sends traffic, generates synthetic ground truth, and reviews monitoring executions in Amazon SageMaker Studio.

### SageMaker has useful built-in features but significant deployment limits
[09:46](https://www.youtube.com/watch?v=5vGWxwf3jr8&t=586s)
The SageMaker demonstration shows scheduled monitor runs, visualizations for drift, real-time data and model-quality monitoring, and a wizard for creating schedules without writing Python. The presenters also describe limits. The tool works with tabular data, does not support endpoints hosting multiple models, and analyzes an inference pipeline as a whole rather than providing separate analysis for each container. It also requires substantial setup, including permissions, storage paths, model deployment, baseline generation, and endpoint configuration. These constraints matter when a system does not already use AWS or when its serving design is more complex.

### Data profiles can reduce the need to send raw production data
[21:32](https://www.youtube.com/watch?v=5vGWxwf3jr8&t=1292s)
Nicholas introduces an open-source logging approach designed to work across cloud vendors. Instead of recording raw inputs, it records data profiles containing descriptive statistics, such as means, medians, and category frequencies. The logger can be packaged with a model served through Flask and can write profiles to a hosted platform or object storage such as Amazon S3. In the demo, training data becomes the baseline profile, while test or endpoint data becomes another profile. The platform compares distributions for individual features and the prediction target, with monitors for distribution drift and missing values. This approach addresses one security concern because the logged object is a summary rather than the original record.

### Evidently AI gives small teams a fast way to create monitoring reports
[32:29](https://www.youtube.com/watch?v=5vGWxwf3jr8&t=1949s)
Alexandre presents Evidently AI as a Python library for data-quality and model monitoring. It compares a reference dataset with current data, which can mean training versus production data or development versus deployed data. The demonstrated reports cover data drift, numerical and categorical target drift, and model performance for regression, classification, and probabilistic classification. Reports can appear interactively in Jupyter notebooks or be exported as HTML and JSON. In the churn example, the report compares training and test data, shows drift by feature, and reports accuracy, precision, recall, F1 score, class representation, confusion matrices, and quality metrics by class. The same reports can run from a scheduled pipeline or command line.

### Tool choice depends on the system already in place
[45:20](https://www.youtube.com/watch?v=5vGWxwf3jr8&t=2720s)
In the closing discussion, the Loka Team avoids presenting one monitoring product as universally best. Amazon SageMaker Model Monitor makes sense when a team already runs inside AWS and can use its related services. A simpler library may fit a small model or a team that wants quick reports with less boilerplate. A cloud-agnostic option may be preferable when models span vendors or need to move between clouds. The presenters also mention differences in features, interfaces, and integration with existing systems. Their conclusion is that personal preference matters after the environment and use case have narrowed the options.

## Notable quotes
- Loka Team: "Model monitoring is the process of closely tracking the performance of machine learning systems in production." (01:32)
- Loka Team: "You need to define and store your baseline." (04:07)
- Loka Team: "The information is very rich and it's extremely easy to generate these reports." (41:20)
- Loka Team: "In the end it came a lot to it depends on exactly what you're doing." (49:40)

## Tools & references mentioned
- Amazon SageMaker
- Amazon SageMaker Model Monitor
- Amazon SageMaker Clarify
- Amazon S3
- AWS
- Evidently AI
- Flask
- Jupyter notebooks
- Great Expectations
- Fiddler
- Arize
- Superwise
- MLflow
- Apache Airflow
- SageMaker Pipelines
- SageMaker Autopilot

## Who should watch
- You are responsible for a deployed model and currently monitor only service uptime or latency.
- Your team needs to choose between a cloud-managed monitoring service and a Python-based or cloud-agnostic approach.
- You want a concrete starting workflow for baselines, drift checks, reports, and alerts using a customer-churn example.

## Related talks

- [Model Watching: Keeping Your Project in Production](https://mlopstalks.com/talks/model-watching-keeping-your-project-in-production) (Ben Wilson, Databricks, 53:08)
- [Model Performance Monitoring and Why You Need it Yesterday](https://mlopstalks.com/talks/model-performance-monitoring-and-why-you-need-it-yesterday) (Amit Paka, Fiddler AI, 1:06:51)
- [The Not So Talked About Reasons Model Monitoring Fails](https://mlopstalks.com/talks/the-not-so-talked-about-reasons-model-monitoring-fails) (Oren Razon, Superwise, 56:03)
- [Model Monitoring in Practice: Top Trends](https://mlopstalks.com/talks/model-monitoring-in-practice-top-trends) (Krishnaram Kenthapadi, Fiddler AI, 51:34)
- [ML Observability](https://mlopstalks.com/talks/ml-observability) (Aparna Dhinakaran, Arize AI, 55:04)