# Monitoring Unstructured Data

Aparna Dhinakaran & Jason Lopatecki, Arize AI | MLOps Lightning Sessions | Episode 2 | 13:12

Source: https://www.youtube.com/watch?v=8Bx7ccrtabk
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/monitoring-unstructured-data
Published: 2022-09-27
Tags: data-quality, drift, monitoring

## TL;DR
- Unstructured production data can differ sharply from the clean data used to train a model, and teams often cannot tell when that change happens.
- Embeddings are vector representations that preserve relationships in words, images, audio, and other unstructured data, so they expose information about what a model learned.
- Arize monitors embedding drift, finds problematic groups and similar production examples, and lets teams export those samples for labeling or data-quality work.

## Summary
Jason Lopatecki and Aparna Dhinakaran explain why monitoring unstructured data is difficult. Production inputs can change without the team noticing, and the data is often unlabeled, so a model's decisions are hard to judge. A model trained on clean English text may encounter Spanish text or other unfamiliar inputs in production. Aparna explains embeddings as vector representations that compress data while preserving relationships between similar items. They can represent words, images, audio, and larger unstructured inputs. Arize uses embeddings as an interface for observing these models. Its workflow collects and indexes production embeddings, measures drift against a baseline with distances such as cosine or Euclidean distance, and surfaces groups that changed. The demo uses customer reviews. A new Spanish-language group appears in production, and a 3D visualization helps the team inspect the affected points. Those samples can then be exported for labeling or data-quality fixes.

## Key ideas
### Unstructured production data can change without the team noticing
[01:09](https://www.youtube.com/watch?v=8Bx7ccrtabk&t=69s)
Jason Lopatecki says much of the data teams handle in production is unstructured, and some of it is unlabeled. That makes it difficult to know whether a model is behaving correctly. The production environment can also differ from training data, which may contain clean, well-formed English words. A deployed model may instead receive Spanish text or other inputs that look very different. When that happens, performance can drop while the team has little direct evidence about why.

### Embeddings expose what complex models have learned
[03:02](https://www.youtube.com/watch?v=8Bx7ccrtabk&t=182s)
Aparna Dhinakaran describes embeddings as internal representations learned by models. They appear in systems such as GPT-3 and DALL-E 2, and they support computer vision and natural language tasks. An embedding is a mathematical vector that compresses data while preserving relationships. Similar words have vectors that are closer together, while dissimilar words are farther apart. The same idea can represent images, audio signals, and other large unstructured inputs.

### Embeddings can be more useful than pixel-level explanations for troubleshooting
[05:03](https://www.youtube.com/watch?v=8Bx7ccrtabk&t=303s)
For a panda classifier, an explainability method such as integrated gradients can show which pixels influenced the prediction. Aparna says that information may not explain the underlying reason for the model's decision. An embedding can contain higher-level structure, such as whether the image has features similar to a bear or what color the fur and face are. That structure gives a data scientist or ML engineer another way to investigate failures in unstructured data.

### Arize monitors every production embedding and surfaces problematic groups
[06:31](https://www.youtube.com/watch?v=8Bx7ccrtabk&t=391s)
Arize's unstructured-data workflow collects, monitors, and indexes each embedding from production inference. It looks for patterns of change across the embeddings and surfaces problematic segments. If Spanish text begins appearing in production, the system can identify that group, find similar examples across the collected production data, and export the samples. The data can go to a labeling provider or back to the data team for investigation.

### Embedding drift compares production vectors with a baseline
[07:58](https://www.youtube.com/watch?v=8Bx7ccrtabk&t=478s)
Aparna explains embedding drift as a comparison between a baseline group of vectors and the vectors seen in production. The distance can be measured with Euclidean distance or cosine distance in a high-dimensional space. When drift is detected, users can open interactive visualizations and inspect the groups causing the change. A group that appears only in production may show that the model is now receiving a type of data absent from training.

### The demo finds a Spanish-language group in customer reviews
[09:54](https://www.youtube.com/watch?v=8Bx7ccrtabk&t=594s)
The demo uses a model that processes e-commerce customer reviews. Arize shows structured fields alongside an embedding-based text vector, with the vector monitored for drift. Earlier periods contain broad positive and negative review groups. A spike appears around mid-June, so Aparna zooms into the affected time window with a 3D UMAP visualization. The separate cluster turns out to contain Spanish reviews that were not present in training.

### Exporting production examples connects monitoring to data work
[09:09](https://www.youtube.com/watch?v=8Bx7ccrtabk&t=549s)
The workflow does more than report that a vector changed. It helps the team inspect the affected data, identify outliers and similar examples, and export samples. Those examples can support better labeling or data-quality work. Aparna presents this as a way to use production data to find what changed and then improve the data used by the model.

## Notable quotes
- Aparna Dhinakaran: "Embeddings are everywhere." (03:02)
- Aparna Dhinakaran: "Embeddings are the internal representation of what the models learn." (03:25)
- Aparna Dhinakaran: "What we do is collect, monitor and index every single embedding." (06:54)
- Aparna Dhinakaran: "It is really clear once I go look at the raw data, these are my Spanish data points that are showing up in production, wasn't there in training." (12:29)

## Tools & references mentioned
- Arize
- Arize AI
- GPT-3
- DALL-E 2
- OpenAI
- integrated gradients
- UMAP

## Who should watch
- You run computer vision, NLP, or another model that receives images, text, audio, or other unstructured inputs in production.
- Your production data is unlabeled, changes over time, or differs from the clean data used during training.
- You need to connect embedding drift alerts with concrete examples that a data or labeling team can inspect.

## Editor's note

From the pack [Monitoring and drift](https://mlopstalks.com/packs/monitoring-and-drift):

Gar needs the training-data version that belongs to the deployed model, and Paka describes how missing model records made comparisons hard to trust. ZenML records pipeline steps, inputs, outputs and code versions for each run, giving investigations a record of how a model was produced. Live request capture, outcome joins and alert thresholds still need to be designed for the service being monitored.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Visualize - Bringing Structure to Unstructured Data](https://mlopstalks.com/talks/visualize-bringing-structure-to-unstructured-data) (Markus Stoll, Renumics, 50:39)
- [Fixing Your ML Data Blind Spots](https://mlopstalks.com/talks/fixing-your-ml-data-blind-spots) (Yash Sheth, Galileo, 51:41)
- [Beyond the Hype: Monitoring LLMs in Production](https://mlopstalks.com/talks/beyond-the-hype-monitoring-llms-in-production) (Claire Longo, Arize, 16:47)
- [Bringing Structure to Unstructured Data with an AI-First System Design](https://mlopstalks.com/talks/bringing-structure-to-unstructured-data-with-an-ai-first-system-design) (Will Gaviria Rojas, Coactive AI, 12:16)
- [Better Use Cases for Text Embeddings](https://mlopstalks.com/talks/better-use-cases-for-text-embeddings) (Vincent Warmerdam, Explosion, 48:20)