# Visualize - Bringing Structure to Unstructured Data

Markus Stoll, Renumics | MLOps Podcast | Episode 258 | 50:39
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=HN8o4D7UjAE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/visualize-bringing-structure-to-unstructured-data
Published: 2024-09-03
Tags: embeddings, evals, multimodal, rag

## TL;DR
- Markus Stoll uses UMAP to project embeddings into two dimensions, where clusters, anomalies, document coverage, and query usage become easier to inspect.
- Renumics Spotlight links embeddings with text, images, audio, video, 3D data, and metadata so ML engineers and domain experts can examine the same examples.
- For messy industrial data, Markus Stoll recommends starting with one specific use case, building a simple first version, and iterating instead of trying to create a perfect dataset upfront.

## Summary
Markus Stoll explains how Renumics uses embeddings and visualization to make large, messy ML datasets easier to inspect. He prefers two-dimensional UMAP maps because they are easier to understand, present, and discuss than 3D plots. In RAG projects, teams can place documents and reference questions in the same embedding space, color documents by query usage, and compare coverage with user feedback. Renumics Spotlight applies the same approach to text, images, audio, video, and 3D simulation data. Markus also describes industrial use cases, including grouping similar car-crash simulations, detecting errors in test-drive data, and classifying connections between simulation parts. The practical obstacles are substantial. Sensor streams come from different systems, signals need synchronization, labels are expensive, and older datasets may use inconsistent channel names. His approach is to clean only what a specific use case needs, release a simple first version, observe how customers use it, and then improve the data and system.

## Key ideas
### UMAP makes embedding spaces easier to inspect in two dimensions
[02:29](https://www.youtube.com/watch?v=HN8o4D7UjAE&t=149s)
Markus Stoll describes reducing document embeddings to a two-dimensional map with UMAP. The map can reveal topic clusters and unexpected groups of points. In one example, Wikipedia header and footer text had entered a database and appeared as small clusters that he could remove after seeing the visualization. He prefers 2D over 3D because a flat image is easier to present on slides and discuss with a team. He uses the map to inspect points outside the main clusters, then asks whether they are useful data or anomalies. The map gives the team a way to look at individual records while still seeing the overall structure of the dataset.

### Reference questions show which parts of a RAG corpus are actually covered
[06:14](https://www.youtube.com/watch?v=HN8o4D7UjAE&t=374s)
Markus explains that questions can be embedded into the same space as documents. The nearest documents to a question indicate which content may be relevant, and the next five or twenty neighbors may contribute to an answer. Teams can color documents by how often questions reference them, creating a heat map of usage across the embedding space. When reference questions cover only a small region, Markus says the team should discuss whether important questions are missing, whether some documents are unnecessary, or whether the corpus needs to change. He considers this coverage check an efficient way to clarify what the customer expects before optimizing the system.

### User feedback is more useful than relying only on automatic RAG metrics
[10:57](https://www.youtube.com/watch?v=HN8o4D7UjAE&t=657s)
Markus wants projects to collect feedback from users early. A simple good-or-bad response, followed by discussion of a questionable answer, can be mapped back onto document clusters. This lets the team see which areas of the corpus produce good or poor results, rather than only seeing which documents are retrieved. He is cautious about deciding quality from automatic measurements alone and says real user feedback matters more for understanding whether the answers work in the intended setting. The same visual analysis can compare customer reference questions with the questions end users ask later, showing where the evaluation set does not match actual usage.

### Renumics Spotlight connects embedding maps to the underlying examples
[12:52](https://www.youtube.com/watch?v=HN8o4D7UjAE&t=772s)
Markus describes Renumics Spotlight as an open-source tool for exploring machine-learning data across text, images, video, audio, 3D data, and embeddings. A similarity map gives users an overview, while selecting a point lets them inspect the underlying item in detail. For simulation data, a point can open the actual 3D car-crash animation, so a simulation engineer and an ML engineer can examine the same result from different views. The goal is to combine the statistical or embedding-based view of a dataset with a close inspection of each example. Markus also mentions a simple vanilla RAG project that can visualize document and question embeddings.

### Simulation embeddings can group similar failures and unusual results
[17:02](https://www.youtube.com/watch?v=HN8o4D7UjAE&t=1022s)
For car-crash simulations, Markus says teams can define a region of interest, such as a selected area of a mesh, rather than embedding the entire animation and vehicle. Embeddings from that region can be projected with UMAP to group simulations with similar deformation or defects. Outliers may point to unusual configurations or results that deserve closer inspection. He presents this as a way to compare simulation outcomes while retaining access to the original 3D animation. Renumics also worked on a classifier that detected how parts in a simulation should be connected. Engineers could use the predictions to create many connections automatically and then review the results.

### Industrial test data needs anomaly detection across many separate modalities
[22:43](https://www.youtube.com/watch?v=HN8o4D7UjAE&t=1363s)
Markus describes car test drives as a source of sensor, audio, image, and other time-series data. Historical examples can support classifiers for errors that have already been seen. When labeled examples are missing, anomaly detection can find rare cases for deeper review. He uses embeddings from different signals to create similarity maps, then compares clusters across modalities. The visualizations can be separate but linked, so selecting an audio cluster also reveals corresponding groups in sensor data. A basic early check can catch setup problems, such as sensors connected incorrectly or cables switched, before the team spends time interpreting bad measurements.

### Natural-language queries should lead to data views, not only text answers
[30:49](https://www.youtube.com/watch?v=HN8o4D7UjAE&t=1849s)
Markus describes combining natural-language questions with tabular and time-series test data. A question such as "Which was my best round on the test drive?" could produce an SQL statement that finds the fastest round. A harder question about an unusual sound could retrieve the relevant records and channels. He says the result should eventually include a visualization of the channels related to the event, a highlighted time range, and the item's location in the UMAP map. A text response alone is less useful for a test driver or engineer who needs to inspect the underlying measurements. The aim is to make this data accessible without requiring every user to write queries.

### Different modalities and industrial domains often require different models
[33:44](https://www.youtube.com/watch?v=HN8o4D7UjAE&t=2024s)
Markus says Renumics currently uses separate models and embeddings for audio, images, video, and other signals instead of forcing every modality into one model. Internet-trained models often do not fit industrial data. A model trained on cats, dogs, and people may be poorly suited to production-machine images or specialized sensor streams. Depending on the project, the team may fine-tune an existing model or train one from scratch. The classifier for simulation-part connections was trained without initial weights. Markus does not claim that one strategy always works. The data, labels, and target task determine whether pretrained models are sufficient.

### A narrow first use case is better than trying to perfect the whole dataset
[44:06](https://www.youtube.com/watch?v=HN8o4D7UjAE&t=2646s)
Industrial datasets can contain large volumes of low-frequency sensor records, inconsistent channel names, separate storage systems, synchronization drift, and expensive labeling work. Markus advises cleaning only the data needed for a specific classification or anomaly-detection task. Trying to make the entire dataset perfect before using it can consume time without clarifying what the final use case requires. His team starts RAG projects with a simple vanilla system that can be built in days or weeks, then watches the questions customers ask and improves the system from there. He applies the same short feedback loop to data preparation: build something usable, find the actual problem, and fix that problem first.

## Notable quotes
- Markus Stoll: "I prefer to use a 2D presentation because that is a little bit easier to overview." (04:54)
- Markus Stoll: "We always try to include very early in our projects some feedback mechanism for the users." (12:25)
- Markus Stoll: "I don't think it's a good idea to start with cleaning up the data until it's perfect." (44:49)
- Markus Stoll: "We really start with a vanilla RAG." (47:13)

## Tools & references mentioned
- Renumics
- Renumics Spotlight
- UMAP
- RAG
- Towards Data Science
- Wikipedia
- SQL
- SAS Decision Builder
- Microsoft Fabric

## Who should watch
- You are building a RAG system and need to understand which documents, questions, and answer failures are represented in your evaluation data.
- Your ML dataset combines text, images, audio, video, sensor streams, or 3D simulations, and you need a way to inspect clusters and individual examples together.
- You work with industrial or test-drive data and want to start with a focused classifier or anomaly detector before spending heavily on full-dataset cleanup.

## Related talks

- [Monitoring Unstructured Data](https://mlopstalks.com/talks/monitoring-unstructured-data) (Aparna Dhinakaran & Jason Lopatecki, Arize AI, 13:12)
- [Bringing Structure to Unstructured Data with an AI-First System Design](https://mlopstalks.com/talks/bringing-structure-to-unstructured-data-with-an-ai-first-system-design) (Will Gaviria Rojas, Coactive AI, 12:16)
- [Model Blind Spot Discovery for Better Models](https://mlopstalks.com/talks/model-blind-spot-discovery-for-better-models) (Pavol Bielik, LatticeFlow & David Garnitz, VectorFlow, 55:35)
- [Better Use Cases for Text Embeddings](https://mlopstalks.com/talks/better-use-cases-for-text-embeddings) (Vincent Warmerdam, Explosion, 48:20)
- [Accelerating Multimodal AI](https://mlopstalks.com/talks/accelerating-multimodal-ai) (Ethan Rosenthal, Runway, 54:58)