Data-centric AI focuses on improving data to solve machine learning problems, while MLOps covers the wider operation of models, data, infrastructure, and parallelization.
2
Cleanlab detects label errors by comparing labels with model-predicted probabilities and estimating the noise process that produced the observed data.
3
Automated data cleaning still needs a human in the loop, with Cleanlab ranking likely errors so people can review the most important cases first.
Summary
Curtis Northcutt explains how his work on cheating detection for MIT and Harvard online courses led to Cleanlab. The cheating problem was a noisy-label problem because the system had examples of cheating but could not reliably identify all the people who had not cheated. Northcutt developed methods for positive-unlabeled learning, noisy-label classification, and confident learning. Cleanlab uses model predictions and noisy labels to estimate false-positive and false-negative rates, find likely label errors, and rank examples by quality. It works with any model that produces predictions and can be used through Python functions or a scikit-learn-compatible class. Northcutt is clear that automation does not remove the need for people. The open-source package is prescriptive, while Cleanlab Studio also helps users review and correct data. He connects the method to noisy-channel ideas from quantum computing and describes applications at Facebook, Amazon, Oculus, and Google. He also argues that vague AI product marketing makes it hard for customers to understand what they are paying for.
Data-centric AI treats data as part of the solution to machine learning problems
Curtis Northcutt describes MLOps as a broad industry term covering models, data, infrastructure, and parallelization. Data-centric AI focuses more specifically on fixing data so the AI pipeline performs better. He says that, with noisy data and noisy labels, methods that alter the data can outperform methods that alter the model. He uses k-nearest neighbors to make the idea concrete: the prediction comes directly from nearby data and their labels, without a large model doing the work.
Cheating detection exposed the problem of missing and incorrect labels
Northcutt's path began with the Cameo cheating detection system for MIT and Harvard online courses. Students created multiple edX accounts, used one account to reveal answers, and submitted them from another account. Simple filters based on repeated IP addresses and close timing found some cases, but strict thresholds produced many false negatives. He realised that cheating detection was a noisy-label problem. The system knew some positive examples, such as students who cheated, but could not reliably establish all the negative examples. That motivated six years of work on machine learning with messy human-generated data.
Positive-unlabeled learning handles cases where only one side of the label is reliable
Curtis explains positive-unlabeled learning through the asymmetry between knowing that someone cheated and knowing that someone did not cheat. In a cheating dataset, confirmed cheating cases are positive labels, while the remaining examples are not necessarily confirmed negatives. He compares this with the legal principle that someone is presumed innocent until proven guilty. His early work extended this idea from positive-unlabeled learning to binary classification with both false positives and false negatives, including the rank pruning paper.
The same label-error problem appears at very large companies
At Facebook, Northcutt worked on bias in upvotes and downvotes for comment rankings. A user might upvote offensive content because it came from a political group they support, which makes the vote a potentially unreliable label. At Amazon, he worked on Alexa's wake-word problem. Amazon could observe when a device woke up, but it could not directly observe every time a device failed to wake up because the keyword was missed. Cleanlab estimated the missing error rate from other observable parts of the noise matrix.
Cleanlab estimates noise by separating observed labels from likely true labels
Northcutt describes a joint matrix of noisy labels and true labels. Its off-diagonal values correspond to label-flipping rates, such as examples labelled zero that should be one. The diagonal values describe consistency. If enough parts of the matrix can be estimated from model predictions and high-quality examples, the remaining rate can be inferred because the full matrix sums to one. He also distinguishes model uncertainty from label noise and says assumptions are required to separate the two.
Model probabilities provide an intuitive way to find suspicious examples
A simple example is an item labelled dog that a trained model assigns a very high probability of being cat. That disagreement makes the example suspicious. The difficult part is choosing a threshold for what counts as suspicious. Northcutt mentions approaches based on percentiles, means, and modelling the full joint distribution. Cleanlab takes model-predicted probabilities and existing labels, then returns likely label issues and quality scores.
The package is model-agnostic and fits into existing workflows
Cleanlab does not depend on a particular model architecture or data type. Northcutt says it works with machine learning data when a model can be run on it, and it uses the model's outputs rather than caring which model produced them. Users can pass predicted probabilities and labels into functions, or use the clean learning class with a scikit-learn model. This design lets the system remain useful as model architectures change.
Automation reduces review work but does not replace human judgment
Northcutt rejects the idea that one button can perfectly fix every dataset. The open-source package is prescriptive: it finds issues and describes dataset quality, but it does not change the dataset. Cleanlab Studio adds infrastructure, trains models, suggests corrections, and ranks the data points most likely to be wrong. A person still reviews those suggestions. Northcutt's goal is to make a dataset review take far less time by sending the highest-priority cases to a human first.
Confident learning applies noisy-channel reasoning to datasets
Northcutt connects Cleanlab to NMR quantum computing, where researchers estimate a noisy channel that separates the true computation from what the machine observed. He and Isaac Chuang applied the same reasoning to data. The observed dataset contains label errors, out-of-distribution examples, and other noise. By estimating the process that created the noisy observations, Cleanlab can infer likely true labels, identify problematic annotators, and help train models on cleaner data.
"If you want to actually train AI machine learning models on really messy noisy label data especially for human-centric problems, people's data is messy and people's data is really noisy."Curtis Northcutt11:41
Who should watch
You are building machine learning systems on labelled data and need a practical way to find likely annotation errors.
Your dataset has missing negatives, inconsistent annotators, or labels that are expensive to verify manually.
You want to understand where data-centric AI fits alongside MLOps and how Cleanlab uses model outputs without depending on a specific model architecture.
