# Monitoring and drift

A pack of 8 sessions from the MLOps Community YouTube channel, in the order to watch them. 6h 21m of video.
Page: https://mlopstalks.com/packs/monitoring-and-drift

The endpoint is healthy, yet recommendations are empty or one customer group receives worse predictions. Monitoring needs to reveal those failures without paging someone for every change in a distribution. Start with a useful-response metric at the service boundary, then capture the inputs the model actually used and connect them to the right training baseline. Keep the customer groups from validation visible in production. Separate broken data from changing inputs and changing relationships before investigating a specific population or a new cluster of text. The final talks turn observations into a manageable alerting practice: use meaningful windows, match urgency to business impact, and send enough context to someone who can investigate. A drift score starts that investigation; it does not explain the cause or choose the remedy.

## This pack is for you if

- Infrastructure dashboards are green while users receive empty or unhelpful predictions.
- Your drift alerts fire often, but nobody can tell whether model performance is actually worse.
- Labels arrive weeks later, and you need earlier evidence that production inputs have changed.

## The talks, in order

### 1. Monitoring the Machine Learning Stack

Lina Weichbrodt, DKB | 55:32 | MLOps Meetup
Video: https://www.youtube.com/watch?v=Un30yb1WlpU
Summary: https://mlopstalks.com/talks/monitoring-the-machine-learning-stack.md

Why first: Weichbrodt's service returned successful responses while its recommendations were wrong or empty. A simple useful-response condition exposed what ordinary uptime checks missed. Start with that observable failure before building a dashboard of model statistics whose connection to users is less clear.

### 2. How Data Capture Transforms ML Observability

Pushkar Gar, Clari | 23:49 | MLOps Community
Video: https://www.youtube.com/watch?v=LHNt6up1c98
Summary: https://mlopstalks.com/talks/how-data-capture-transforms-ml-observability.md

Why second: Gar shows why logging a request may still miss the data needed for diagnosis. Features fetched inside the model must be captured too, and the baseline must match the deployed model's training-data version. His endpoint buffer makes the evidence behind the response available for the comparisons that follow.

### 3. Model Performance Monitoring and Why You Need it Yesterday

Amit Paka, Fiddler AI | 1:06:51 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=mesTMiFQcTU
Summary: https://mlopstalks.com/talks/model-performance-monitoring-and-why-you-need-it-yesterday.md

Why here: Paka carries validation's important customer groups into live monitoring. The same slices should remain visible after launch, with leading data signals when outcomes arrive late. This part of his 2021 account connects captured traffic to the expectations you already established, rather than replacing them with a global average.

### 4. ML Drift: How to Identify Issues Before They Become Problems

Amy Hodler, Fiddler | 56:58 | MLOps Meetup
Video: https://www.youtube.com/watch?v=--KcBoInuqw
Summary: https://mlopstalks.com/talks/ml-drift-how-to-identify-issues-before-they-become-problems.md

Why here: Hodler distinguishes a change in inputs from a changed relationship to outcomes, and shows how swapped fields or wrong units can resemble either problem. Which signal you can trust depends on whether labels are available. That distinction keeps an early warning from becoming an unsupported diagnosis of model decay.

### 5. ML Observability

Aparna Dhinakaran, Arize AI | 55:04 | MLOps Meetup
Video: https://www.youtube.com/watch?v=joTF9BRwWp4
Summary: https://mlopstalks.com/talks/ml-observability.md

Why here: Dhinakaran's 2020 fraud-model demo follows one complaint down to a specific population and a previously unseen feature value. Comparing predictions, actual outcomes and reference distributions narrows the investigation. It is a concrete example of using the distinctions above instead of stopping at the alert.

### 6. Monitoring Unstructured Data

Aparna Dhinakaran & Jason Lopatecki, Arize AI | 13:12 | MLOps Lightning Sessions
Video: https://www.youtube.com/watch?v=8Bx7ccrtabk
Summary: https://mlopstalks.com/talks/monitoring-unstructured-data.md

Why here: Text and images need another way to locate changing inputs. Dhinakaran and Lopatecki's embedding demonstration finds a new group of Spanish reviews, then inspects the actual examples. The vector comparison guides attention; reading the changed data explains what the cluster means.

### 7. Model Watching: Keeping Your Project in Production

Ben Wilson, Databricks | 53:08 | MLOps Meetup
Video: https://www.youtube.com/watch?v=7HJ5x-DglLE
Summary: https://mlopstalks.com/talks/model-watching-keeping-your-project-in-production.md

Why here: Wilson puts those comparisons on a sensible schedule and severity scale. Rolling windows describe sustained changes in inputs and predictions, while the business cost determines how urgently someone should respond. An interesting shift in an experiment need not create the same page as a failing critical model.

### 8. The Not So Talked About Reasons Model Monitoring Fails

Oren Razon, Superwise | 56:03 | MLOps Meetup
Video: https://www.youtube.com/watch?v=_UQ5IV73MW4
Summary: https://mlopstalks.com/talks/the-not-so-talked-about-reasons-model-monitoring-fails.md

Why last: Razon asks what happens after a useful signal exists. An output anomaly needs context from inputs, labels and business behavior, then an owner who can use it. Routing different issues through people's existing workflows closes the monitoring problem at an actionable investigation, before deciding whether any model update is needed.

## Editor's note

Gar needs the training-data version that belongs to the deployed model, and Paka describes how missing model records made comparisons hard to trust. ZenML records pipeline steps, inputs, outputs and code versions for each run, giving investigations a record of how a model was produced. Live request capture, outcome joins and alert thresholds still need to be designed for the service being monitored.

Written by the MLOps Talks editors (the ZenML team), not by any of the speakers.