# Data Selection for Data-Centric AI: Data Quality Over Quantity

Cody Coleman | MLOps Coffee Sessions | Episode 59 | 1:11:01

Source: https://www.youtube.com/watch?v=v7Pj7a6KXSU
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/data-selection-for-data-centric-ai-data-quality-over-quantity
Published: 2021-10-11
Tags: benchmarks, data-engineering, data-quality, human-in-the-loop

## TL;DR
- Cody Coleman argues that data should receive at least as much attention as model architecture because poor data creates costs throughout an ML system.
- Active learning can use a model's uncertainty to identify examples that humans should label, reducing guesswork and making smaller, more carefully reviewed datasets possible.
- Practitioners should start with a small, clean data source, inspect data quality at each pipeline stage, and add legacy sources gradually rather than trying to fix an entire organization's data at once.

## Summary
Cody Coleman describes why machine learning should focus more on data quality and selection. He connects this view to his work on DAWNBench and MLPerf, which addressed the cost and time of training and inference, and to his research on active learning and core-set selection. His practical advice starts with cataloging data, establishing a quality baseline, and checking each transformation in the pipeline. Once raw data is reliable, teams can select a smaller set of examples for labeling instead of using every available record. Coleman explains this with a bowling classifier that surfaced confusing images of exercise studios and empty grocery shelves. He also argues that humans remain necessary for defining ambiguous concepts, while automation should direct their attention to the examples that matter. For organizations with old or unreliable systems, he recommends starting with a narrowly scoped model and clean source, then adding other sources incrementally.

## Key ideas
### Computers gave Coleman an accessible route into engineering
[05:41](https://www.youtube.com/watch?v=v7Pj7a6KXSU&t=341s)
Cody Coleman grew up in South Jersey after being born while his mother was in prison and being adopted by his maternal grandparents. His family lived on welfare and food stamps, and his high school ranked around 300 out of 322 schools in New Jersey. Computers felt different from fields such as mechanical or chemical engineering because he could work with a computer and internet connection without expensive equipment. Building a computer with his older brother helped turn technology from something magical into something tangible he could make himself.

### A mentor changed Coleman's view of what was possible
[07:53](https://www.youtube.com/watch?v=v7Pj7a6KXSU&t=473s)
Coleman's older brother returned to his life after finding work in IT without a degree. He helped build Coleman's first computer and encouraged him to consider selective universities. Coleman initially assumed Princeton would not accept someone like him and that he could not afford it. His brother explained that strong academic performance gave him a chance and that financial aid could address the cost. That conversation changed Coleman's mindset from asking why bother to asking why not. Supportive teachers, basic programming classes, and his own persistence helped him reach MIT.

### Data systems shaped Coleman's path into machine learning
[15:12](https://www.youtube.com/watch?v=v7Pj7a6KXSU&t=912s)
Before focusing on machine learning, Coleman worked with data through web development, databases, and educational technology. At MIT, he worked with Sanjay Sarma on educational data mining and improved an edX grader by vectorizing its grading operation. The faster grader made it possible to provide incremental grades and study interventions for students in massive open online courses. Later, at a trading company in Chicago, he built an ETL pipeline and data integrity checks because bad data could affect decisions about financial positions. These experiences showed him both the value of data and the resources required to use machine learning in practice.

### Benchmarks addressed the cost of using machine learning
[22:43](https://www.youtube.com/watch?v=v7Pj7a6KXSU&t=1363s)
Coleman began his PhD focused on practical barriers around model training and inference. He recalls that a single ResNet-50 training run cost about a thousand dollars in cloud credits and could take more than 24 hours on a standard AWS instance at the time. DAWNBench was created to push down training time and cost, then expanded into MLPerf and MLCommons as a broader industry effort. Coleman says benchmarks shape the field, so measuring system performance gave researchers and companies a shared way to improve practical machine learning systems.

### Data should receive the same attention as models
[25:17](https://www.youtube.com/watch?v=v7Pj7a6KXSU&t=1517s)
After seeing work on system performance gain momentum, Coleman turned to another barrier: data. He says models were treated as the heroes of deep learning while data was treated as a necessary nuisance. His research asked whether teams could use fewer examples while reaching the same performance by selecting data intelligently. This could reduce computation, labeling, and other operational costs. His work included Selection via Proxy and Similarity Search for Efficient Active Learning and Search of Rare Concepts. He describes active learning and core-set selection as ways to make dataset creation less ad hoc.

### Data quality checks belong at every pipeline stage
[33:41](https://www.youtube.com/watch?v=v7Pj7a6KXSU&t=2021s)
Coleman recommends making data central from the start of an ML project. Teams should understand how data is stored and cataloged, identify whether it is structured, semi-structured, or unstructured, and establish a quality baseline. They should inspect data after each processing and ETL step instead of waiting for downstream failures. Coleman says bad data at the beginning has an amplifying effect as it moves through later systems. He mentions Great Expectations as an example of a tool that can help with data quality checks, while noting that unstructured data such as text, images, video, and speech still has less mature tooling.

### Active learning finds examples people may not think to label
[45:06](https://www.youtube.com/watch?v=v7Pj7a6KXSU&t=2706s)
Coleman describes using uncertainty sampling and entropy for a simple bowling classifier. The model selected images of exercise studios because wood floors and colorful exercise balls resembled features of a bowling alley. It also found images of empty grocery shelves during the early pandemic because shelf dividers could look like bowling lanes. These were examples Coleman would not have chosen from his own assumptions about bowling. Letting the model identify uncertain cases exposed possible undefined behavior before it became a production problem. The same approach can surface unexpected errors caused by changes in the world.

### Humans define meaning while automation directs their attention
[58:00](https://www.youtube.com/watch?v=v7Pj7a6KXSU&t=3480s)
Coleman supports automating basic data quality checks and selecting which examples deserve attention, but he expects humans to remain involved. Unstructured data carries context that people can often judge quickly, and business meanings can be ambiguous. He uses the word torch as an example: it can mean a flame, a flashlight in the UK, or PyTorch to an ML researcher. Humans still need to decide what a label means for a particular use case. Active learning can reduce the dataset from millions of examples to hundreds, while tools such as Snorkel can let people encode labeling rules.

### Legacy data projects should begin with a small clean slice
[1:05:03](https://www.youtube.com/watch?v=v7Pj7a6KXSU&t=3903s)
For organizations with old systems and unreliable records, Coleman recommends starting small. A team can choose a simple model, one clean data source, and only the amount of data needed to make an accurate prediction. It can then improve the quality of that source before adding more systems. He compares this to working with legacy software by building a small, clean service and connecting it to older components through an adapter. This approach gives a team a concrete result without requiring the whole organization to fix every historical data problem first.

## Notable quotes
- "Models were basically kind of like the hero of many of deep learning successes, whereas data was always kind of treated as like a necessary evil." (25:17)
- "The more you can do that up front I think the more time it saves you kind of down the road." (36:28)
- "By letting the ML model tell me what it was uncertain about, kind of by using very simple active learning, now I take the guesswork out of that." (47:17)
- "I think fundamentally you need to have a human being to kind of help define that and shape that." (1:00:12)
- "Start with a simple model with a simple clean data source and start with as little data as you need to be able to accurately predict something." (1:05:21)

## Tools & references mentioned
- MLOps Community
- DAWNBench
- MLPerf
- MLCommons
- Selection via Proxy: Efficient Data Selection for Deep Learning
- Similarity Search for Efficient Active Learning and Search of Rare Concepts
- Great Expectations
- Snorkel
- Sanjay Sarma
- Matei Zaharia
- Peter Bailis
- Chris Ré
- Kunle Olukotun
- Andrew Ng
- Andrew Wang
- Google
- Facebook
- Databricks
- Apache Spark
- Mesos
- AWS
- ResNet-50
- edX
- Coursera

## Who should watch
- You are building an ML system from messy legacy data and need a way to begin without repairing every source first.
- Your labeling budget is limited, or your dataset is too large for people to inspect carefully.
- You want to combine automated data checks with human review for ambiguous or changing examples.

## Related talks

- [Look At Your ****ing Data 👀](https://mlopstalks.com/talks/look-at-your-ing-data) (Kenny Daniel, Hyperparam, 1:05:26)
- [Fixing Your ML Data Blind Spots](https://mlopstalks.com/talks/fixing-your-ml-data-blind-spots) (Yash Sheth, Galileo, 51:41)
- [Data Quality = Quality AI](https://mlopstalks.com/talks/data-quality-quality-ai) (Samuel Partee, Redis & Chad Sanderson, Gable & Joe Reis, Ternary Data & Maria Zhang, Proactive AI Lab Inc & Pushkar Garg, Clari Inc, 27:15)
- [Managing Data for Effective GenAI Application](https://mlopstalks.com/talks/managing-data-for-effective-genai-application) (Anu Arora & Anass Bensrhir, QuantumBlack AI by McKinsey, 51:01)
- [Data Engineering + ML + Software Engineering](https://mlopstalks.com/talks/data-engineering-ml-software-engineering) (Satish Chandra Gupta, Slang Labs, 57:05)