# Data-Centric AI Means Centralizing Training Data

Alberto Rizzoli, V7 | MLOps Meetup | Episode 85 | 49:01
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=DFXVIE8GRF8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/data-centric-ai-means-centralizing-training-data
Published: 2021-11-12
Tags: data-engineering, data-quality, human-in-the-loop

## TL;DR
- Alberto Rizzoli argues that training data and its labels increasingly determine model performance, especially when companies work with unique industrial or medical data.
- V7 is built around a continuously changing, versioned training-data library that connects production models, automated labeling, human review, and model retraining.
- Computer vision teams need better annotation standards and integrations so datasets can move between data scientists and MLOps tools without custom conversion work.

## Summary
Alberto Rizzoli argues that computer vision teams are gaining more from improving training data than from repeatedly changing model architectures. Companies need to capture knowledge about their own objects, defects, patients, or documents in a central, queryable dataset. That dataset should change as models run in production: low-confidence examples can return for labeling, models can pre-label new data, and humans can correct or approve the results. V7's product is designed around this loop, with dataset versioning, segmentation, workflow management, model orchestration, and integrations with training tools. Rizzoli also argues that common benchmarks such as ImageNet contain labeling problems that matter little for general research but matter greatly in industry. He calls for test sets organized around real operating conditions, including population, instrument, and environmental differences. He is candid about V7's engineering tradeoffs, including building systems internally and declining 3D lidar to keep its pixel-based product focused.

## Key ideas
### Training data has become a company's private computer vision knowledge
[00:00](https://www.youtube.com/watch?v=DFXVIE8GRF8&t=0s)
Rizzoli says model training used to focus heavily on parameters, handcrafted features, and architecture changes. Faster hardware has shortened many industrial training runs, so the competitive advantage is shifting toward the data a company collects and labels. A solar-panel manufacturer needs examples of its own defects. A medical system needs images tied to particular scanners and populations. A company building fruit, runway, or pathology models develops knowledge about objects that generic datasets do not contain. Rizzoli describes the images and relevant labels as the core value of an AI team's resources. A model cannot learn an object that is barely visible in its training data by changing a hyperparameter or architecture.

### A production model should send useful examples back into the training dataset
[03:58](https://www.youtube.com/watch?v=DFXVIE8GRF8&t=238s)
V7's proposed workflow connects deployment with data collection. A model in production can create several versions and send low-confidence items back to the dataset. The returned data can be raw or pre-labeled, then reviewed by people before it becomes part of a later training run. This closes the loop between what a model does in the cloud or at the edge and what the team knows about its errors. Rizzoli points out that fleets of cars or robots make this difficult because teams must decide what data to upload, when to upload it, and how to make it available to everyone. Versioned datasets, queries, segmentation, and direct handoff to annotators replace folders containing images and attached JSON files.

### Dataset versions need the same treatment as model and parameter versions
[18:37](https://www.youtube.com/watch?v=DFXVIE8GRF8&t=1117s)
Rizzoli wants teams to treat training data as an organized library rather than a pile of files. Data should be searchable and segmented so a team can deliberately select material for training or validation. He compares fixed datasets and changing models with an older research pattern that varies architecture while keeping the data fixed. A better approach changes labels or raw data alongside model parameters, making it possible to study how data quality affects performance. In production, teams may create many dataset versions with different taxonomies, bounding boxes, or class selections. Selecting among those versions is part of experimentation, just as tuning model parameters is.

### Benchmark datasets can hide labeling errors and poor industry fit
[23:00](https://www.youtube.com/watch?v=DFXVIE8GRF8&t=1380s)
Rizzoli uses ImageNet to show why fixed benchmarks deserve scrutiny. Its single-label image classification setup can force a human to choose an aardvark label for an image that may be better described as a shower curtain. He says the aardvark class also contains many anteaters and calls the class 98 percent wrong. That distinction may not affect a general benchmark when the model has no similar animal to confuse it with. Industrial teams face different consequences. A self-driving system may need to distinguish a car from a motorcycle. Rizzoli also criticizes datasets with too many loosely connected labels, since their schemas can become standards without matching the needs of real applications.

### Test sets should describe the conditions that people care about
[26:23](https://www.youtube.com/watch?v=DFXVIE8GRF8&t=1583s)
Large random train, test, and validation splits are not enough for safety-sensitive systems. Rizzoli argues for several test sets covering scenarios that people want to measure directly. A driving model might be checked separately on snow, desert conditions, and debris on the road. Medical models can change performance when the scanner changes, when the imaging machine changes, or when the patient population changes. He gives the example of training on chest X-rays from East Asian populations and testing on people from Southern Europe. Those datasets may show different results for the same pathology. He connects this organization of test data to bias recognition and to making test sets easy to pull into experimentation tracking tools.

### Human review works best when models do the repetitive first pass
[24:16](https://www.youtube.com/watch?v=DFXVIE8GRF8&t=1456s)
Rizzoli describes labeling as a workflow where models perform initial work and people refine or approve it. V7's auto-labeling can produce a human-passable first pass, although it is not always perfect. The company aims to reduce the clicks and fatigue involved in labeling, since a tool that is pleasant to use can help doctors and other specialists produce more labels. Users can also label examples, train a model, and use that model to continue labeling. He says models can be highly accurate even when they are too slow for real-time use on a small GPU. Human approval remains important where a decision carries responsibility, such as purchases or medical work.

### V7 built a changing dataset management system around its own needs
[30:30](https://www.youtube.com/watch?v=DFXVIE8GRF8&t=1830s)
In response to Demetrios Brinkmann's question, Rizzoli explains that much of V7's engineering work went into its database management system and model orchestration. The company chose Elixir to handle real-time changes as labels, assignments, and tags change. A dataset item may trigger a model, which requires a GPU instance, processes the item, and then turns the instance off to save customer money. V7 uses native PyTorch pipelines rather than major cloud-provider training tools. Rizzoli admits that the team sometimes built too much from scratch and could have used more third-party tools early on. He still sees value in controlling the system, while choosing integrations for areas where another company is more focused.

### V7 declined 3D lidar to keep one technical core
[43:49](https://www.youtube.com/watch?v=DFXVIE8GRF8&t=2629s)
Rizzoli says supporting text, audio, images, and lidar too early can produce disconnected products held together by human processes. V7 chose to focus on pixel-based data and to work with third-party tools for 3D lidar. He compares the desired product to Google Search, with one focused system rather than several independent tools. The decision was partly about product focus and partly about avoiding technical debt. Rizzoli says 3D annotation has not yet found an elegant home in the way 3D lidar is produced and labeled. He also argues that the training-data market for document processing may be larger than autonomous driving, even though it attracts less attention because it is less exciting.

## Notable quotes
- Alberto Rizzoli: "The competitive advantage of a lot of companies and one of the reasons being is that every company industry has unique objects that they need to create." (01:34)
- Alberto Rizzoli: "There is no way to make someone learn an object that isn't visible enough in the training data by tweaking hyper parameter or by changing the architecture." (02:57)
- Alberto Rizzoli: "We have a lot of data about and charts and charts in charge about how changing minutely in architecture or adding more layers or adding more parameters to something changes the performance on a fixed data set on ImageNet." (12:08)
- Alberto Rizzoli: "You can't rebuild the ship while it's already in the water." (44:41)
- Alberto Rizzoli: "The best advice is to kind of look at traditional industries and how they're processing a specific piece of data and then AI can just make it faster and more accurate." (41:56)

## Tools & references mentioned
- V7
- MLOps Community
- ImageNet
- Elixir
- PyTorch
- Google Search
- Runway ML
- GANs
- 3D lidar

## Who should watch
- You are building a computer vision system whose data comes from a specific factory, medical device, population, or operating environment.
- Your team has separate labeling, model training, deployment, and experiment tracking steps, and production errors do not flow back into a managed dataset.
- You maintain annotation formats or dataset exports by hand and need a practical view of versioning, human review, and integrations.

## Editor's note

Alberto Rizzoli says teams should treat training data as an organized, searchable library, with production examples returning for review and retraining. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a training artifact or model can be traced to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Data Selection for Data-Centric AI: Data Quality Over Quantity](https://mlopstalks.com/talks/data-selection-for-data-centric-ai-data-quality-over-quantity) (Cody Coleman, 1:11:01)
- [Putting the AI back in Medallion Lake Design](https://mlopstalks.com/talks/putting-the-ai-back-in-medallion-lake-design) (Simon Whiteley, Advancing Analytics, 13:28)
- [How Data Platforms Affect ML & AI](https://mlopstalks.com/talks/how-data-platforms-affect-ml-ai) (Jake Watson, The Oakland Group, 39:12)
- [DataOps and Data Versioning in ML](https://mlopstalks.com/talks/dataops-and-data-versioning-in-ml) (Dmitry Petrov, Iterative.AI, 1:01:55)
- [Accelerating Multimodal AI](https://mlopstalks.com/talks/accelerating-multimodal-ai) (Ethan Rosenthal, Runway, 54:58)