# How Data Capture Transforms ML Observability

Pushkar Gar, Clari | MLOps Community | 23:49

Source: https://www.youtube.com/watch?v=LHNt6up1c98
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-data-capture-transforms-ml-observability
Published: 2024-10-09
Tags: data-quality, drift, monitoring, observability

## TL;DR
- Data capture at the serving endpoint gives teams the inference data needed to check training-serving skew, drift, and other production data problems.
- Managed capture can miss features retrieved from an online feature store and can be awkward for models that return arrays of predictions.
- An in-memory buffer can collect complete prediction inputs and outputs, batch them, and flush them to S3 using configurable limits.

## Summary
Pushkar Gar explains why data capture is the foundation of ML observability. A production ML system can contain production databases, object storage, a data lake, transformed data layers, offline and online feature stores, and model serving endpoints. Quality checks can run at each stage, but the serving endpoint is the last place to identify problems that earlier checks missed. Captured inference data can be compared with versioned training data to detect type changes, missing values, distribution drift, missing columns, and unexpected columns. Pushkar describes limitations in managed capture, including its inability in some cases to record features fetched from an online feature store and its handling of array-like outputs from ranking models. His custom approach adds a buffer inside the model code. The buffer collects data after feature retrieval and prediction, then writes batches to S3 based on memory, message-count, or time limits. Downstream jobs generate metrics, trigger alerts, and support retraining or model-quality checks.

## Key ideas
### The serving endpoint is the final data-quality checkpoint
[02:02](https://www.youtube.com/watch?v=LHNt6up1c98&t=122s)
Pushkar separates the data platform from the ML platform. Data can move from production databases and object storage into a data lake, through bronze, silver, and gold layers, and then into an offline feature store for training. An online feature store can provide current features at prediction time. The serving endpoint brings those inputs together and produces predictions. Checks can run throughout the pipeline, but the endpoint is the last place to catch failures or changes that earlier stages did not record. Data capture at this point provides the material for production monitoring.

### Inference data reveals training-serving skew and concept drift
[05:50](https://www.youtube.com/watch?v=LHNt6up1c98&t=350s)
A model learns relationships from its training data and assigns weights to its features. If production feature distributions differ from the training distributions, the model may need to be retrained on newer data. Pushkar calls this distribution drift. Concept drift is different: the feature distributions may remain similar while the relationship between features and the target changes. He uses COVID as an example of a change in circumstances that could make existing models perform poorly. Capturing inference data allows monitoring jobs to identify both kinds of change.

### Simple column checks provide a practical starting point
[08:50](https://www.youtube.com/watch?v=LHNt6up1c98&t=530s)
Pushkar describes several checks that can run by comparing training and inference data. A type check verifies that each inference column has the same type as its training counterpart. A completeness check compares null counts and can expose a broken pipeline or an unaccounted-for change. A baseline drift check compares the distributions of each column and can trigger an alert after a significant change. Monitoring should also detect missing columns and unexpected extra columns. He mentions DQ as a library that can help compare columns without implementing the algorithms from scratch.

### Training-data versions connect models to the right monitoring baseline
[11:41](https://www.youtube.com/watch?v=LHNt6up1c98&t=701s)
A monitoring job needs both the training data and the inference data. Data capture creates the inference set from requests reaching the endpoint, while the model registry can identify which training-data version belongs to the deployed model. Pushkar says this matters because models go through multiple experiments and A/B tests. The monitoring job can load the matching training data from S3 and use it to create baseline values or constraints, such as expected ranges for feature columns, before checking production data.

### Managed capture can omit features fetched during prediction
[13:30](https://www.youtube.com/watch?v=LHNt6up1c98&t=810s)
In the managed capture flow he describes, a server receives a request, passes it to the model, and persists the request and prediction. This captures what the prediction service sent to the server. It does not necessarily capture features that the model retrieves from an online feature store. If the request contains only a user ID, the stored record may contain the ID and prediction without the current feature values used by the model. One workaround is to use the request time to reconstruct those features from an offline feature store with time-travel queries, but that requires another job.

### A model-side buffer captures the complete prediction context
[19:00](https://www.youtube.com/watch?v=LHNt6up1c98&t=1140s)
Pushkar's custom approach packages a data-capture buffer that data scientists import into model code. After the model retrieves online features and makes a prediction, the model adds the complete plain-text data to the buffer before serializing the response. The buffer uses the model endpoint's memory and compute, then flushes records to S3 according to configurable limits. The implementation can use buffer memory size, message count, and active buffer time as controls. This captures information that a server-level mechanism may not see.

### Downstream monitoring closes the loop with alerts and retraining
[20:29](https://www.youtube.com/watch?v=LHNt6up1c98&t=1229s)
A monitoring job consumes the training data and captured inference data, generates data-quality metrics, and sends them to an observability tool. Thresholds can route alerts to a notification queue, with subscribers sending messages to PagerDuty or Slack. The same captured data can support model-quality monitoring when production records are joined with ground truth. The resulting comparison includes the features, the prediction, and the actual outcome. Retraining can then follow when monitoring identifies a problem.

### Buffer settings and sampling must be measured on the endpoint
[22:00](https://www.youtube.com/watch?v=LHNt6up1c98&t=1320s)
Pushkar recommends batching requests because writing every request to S3 can hurt I/O performance. Teams should test memory limits, message counts, and flush intervals rather than choosing them blindly. Sampling can reduce compute costs when capturing every request is unnecessary. He says that, in his case, capturing 75% of requests was still enough to identify data drift. Since the buffer runs on the serving endpoint, performance benchmarking is also needed to measure its average, maximum, and 99th-percentile latency impact.

## Notable quotes
- "The serving endpoint is sort of the Last Frontier where you can still identify issues in the data pipeline." (08:02)
- "To be able to identify that that has happened in the production data, meaning that there has been a distribution drift in the inference data, we need to be able to capture data." (06:57)
- "You would not want to persist every request that you get to S3 because that hampers your I/O performance." (22:22)
- "It's important to do benchmarking to identify the average, max, 99% high latency that may be added as part of the data capture buffer." (23:21)

## Tools & references mentioned
- MLOps Community
- Clari
- S3
- data lake
- Medallion architecture
- Databricks
- offline feature store
- online feature store
- model registry
- DQ
- SageMaker
- EC2
- Flask
- HTTP
- gRPC
- Protobuf
- PagerDuty
- Slack

## Who should watch
- You operate models whose production features come from an online feature store and need to know exactly what reached the model.
- Your managed serving platform captures requests, but it does not preserve retrieved features or support the shape of your model's output.
- You are designing endpoint monitoring and need practical advice on batching, sampling, buffer limits, and latency tests.

## Editor's note

Pushkar Gar says the serving endpoint is the last place to catch production data problems that earlier checks missed. ZenML records each run's inputs, outputs, steps, and code version, so the inference data can be traced alongside the training data and the model that used it. Its model registry also records lineage for deployed models.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [ML Observability](https://mlopstalks.com/talks/ml-observability) (Aparna Dhinakaran, Arize AI, 55:04)
- [Monitoring Unstructured Data](https://mlopstalks.com/talks/monitoring-unstructured-data) (Aparna Dhinakaran & Jason Lopatecki, Arize AI, 13:12)
- [From Idea to Production ML](https://mlopstalks.com/talks/from-idea-to-production-ml) (Lex Beattie, Spotify, 53:18)
- [Towards Observability for ML Pipelines](https://mlopstalks.com/talks/towards-observability-for-ml-pipelines) (Shreya Shankar, UC Berkeley, 57:10)
- [Mid-Scale Production Feature Engineering](https://mlopstalks.com/talks/mid-scale-production-feature-engineering) (Dr. Venkata Pingali, Scribble Data, 1:01:35)