Unstructured production data can differ sharply from the clean data used to train a model, and teams often cannot tell when that change happens.
2
Embeddings are vector representations that preserve relationships in words, images, audio, and other unstructured data, so they expose information about what a model learned.
3
Arize monitors embedding drift, finds problematic groups and similar production examples, and lets teams export those samples for labeling or data-quality work.
Summary
Jason Lopatecki and Aparna Dhinakaran explain why monitoring unstructured data is difficult. Production inputs can change without the team noticing, and the data is often unlabeled, so a model's decisions are hard to judge. A model trained on clean English text may encounter Spanish text or other unfamiliar inputs in production. Aparna explains embeddings as vector representations that compress data while preserving relationships between similar items. They can represent words, images, audio, and larger unstructured inputs. Arize uses embeddings as an interface for observing these models. Its workflow collects and indexes production embeddings, measures drift against a baseline with distances such as cosine or Euclidean distance, and surfaces groups that changed. The demo uses customer reviews. A new Spanish-language group appears in production, and a 3D visualization helps the team inspect the affected points. Those samples can then be exported for labeling or data-quality fixes.
Unstructured production data can change without the team noticing
Jason Lopatecki says much of the data teams handle in production is unstructured, and some of it is unlabeled. That makes it difficult to know whether a model is behaving correctly. The production environment can also differ from training data, which may contain clean, well-formed English words. A deployed model may instead receive Spanish text or other inputs that look very different. When that happens, performance can drop while the team has little direct evidence about why.
Embeddings expose what complex models have learned
Aparna Dhinakaran describes embeddings as internal representations learned by models. They appear in systems such as GPT-3 and DALL-E 2, and they support computer vision and natural language tasks. An embedding is a mathematical vector that compresses data while preserving relationships. Similar words have vectors that are closer together, while dissimilar words are farther apart. The same idea can represent images, audio signals, and other large unstructured inputs.
Embeddings can be more useful than pixel-level explanations for troubleshooting
For a panda classifier, an explainability method such as integrated gradients can show which pixels influenced the prediction. Aparna says that information may not explain the underlying reason for the model's decision. An embedding can contain higher-level structure, such as whether the image has features similar to a bear or what color the fur and face are. That structure gives a data scientist or ML engineer another way to investigate failures in unstructured data.
Arize monitors every production embedding and surfaces problematic groups
Arize's unstructured-data workflow collects, monitors, and indexes each embedding from production inference. It looks for patterns of change across the embeddings and surfaces problematic segments. If Spanish text begins appearing in production, the system can identify that group, find similar examples across the collected production data, and export the samples. The data can go to a labeling provider or back to the data team for investigation.
Embedding drift compares production vectors with a baseline
Aparna explains embedding drift as a comparison between a baseline group of vectors and the vectors seen in production. The distance can be measured with Euclidean distance or cosine distance in a high-dimensional space. When drift is detected, users can open interactive visualizations and inspect the groups causing the change. A group that appears only in production may show that the model is now receiving a type of data absent from training.
The demo finds a Spanish-language group in customer reviews
The demo uses a model that processes e-commerce customer reviews. Arize shows structured fields alongside an embedding-based text vector, with the vector monitored for drift. Earlier periods contain broad positive and negative review groups. A spike appears around mid-June, so Aparna zooms into the affected time window with a 3D UMAP visualization. The separate cluster turns out to contain Spanish reviews that were not present in training.
Exporting production examples connects monitoring to data work
The workflow does more than report that a vector changed. It helps the team inspect the affected data, identify outliers and similar examples, and export samples. Those examples can support better labeling or data-quality work. Aparna presents this as a way to use production data to find what changed and then improve the data used by the model.
"It is really clear once I go look at the raw data, these are my Spanish data points that are showing up in production, wasn't there in training."Aparna Dhinakaran12:29
Who should watch
You run computer vision, NLP, or another model that receives images, text, audio, or other unstructured inputs in production.
Your production data is unlabeled, changes over time, or differs from the clean data used during training.
You need to connect embedding drift alerts with concrete examples that a data or labeling team can inspect.
