Data curation improves model accuracy by removing harmful training data and selecting the best subset that fits the compute budget.
2
GPU acceleration makes fuzzy deduplication, semantic deduplication, embedding creation, clustering, and classifier inference practical at very large data volumes.
3
NeMo Curator provides open-source tools for distributed, GPU-accelerated data curation across text, images, and other modalities.
Summary
Ryan explains why data curation is necessary when foundation models train on enormous datasets. A curation pipeline starts with raw sources, extracts usable text, applies simple cleaning and filters, then performs deduplication and classifier-based annotation. The expensive stages become difficult when datasets reach hundreds of terabytes or billions of documents. Ryan describes GPU implementations for fuzzy deduplication with MinHash, locality-sensitive hashing, and connected components. He also covers semantic deduplication, which compares embeddings after clustering them to avoid an impossible all-pairs comparison. Classifier inference can be improved by changing batch sizes according to sequence length and using GPU tokenization. The talk focuses on the engineering tradeoffs behind scaling these methods, including approximate comparisons and business-dependent definitions of duplication. Ryan presents NeMo Curator as the open-source home for the tools his team developed.
Curation improves models by removing harmful data and choosing better training examples
Ryan says the success of data curation is measured by downstream model accuracy. Curation helps in two ways: it removes data that is actively harmful during training, and it selects the best subset when there is more data than the compute budget allows. Repeating tokens is usually less useful than seeing new tokens, so deduplication can improve the data mix while reducing wasted training. The pipeline should therefore spend its most expensive processing on the smaller set of documents that survives earlier filters.
A curation pipeline moves from cheap filters to expensive models
The pipeline begins with raw data from cloud storage, a workstation, or a cluster. For web data such as Common Crawl, it extracts plain text from HTML, fixes encoding problems, identifies languages, and applies preliminary quality filters. Simple heuristics can remove repetitive documents or text with excessive punctuation. Later stages perform exact or fuzzy deduplication, semantic deduplication, and annotation with larger classifiers. Ryan says this ordering matters because datasets can contain hundreds of terabytes, making it impractical to run heavy models over everything.
Fuzzy deduplication uses MinHash and locality-sensitive hashing to avoid all-pairs comparison
Fuzzy deduplication catches documents that are almost the same, such as legal documents sharing boilerplate with a few names changed. Each document receives a MinHash signature containing hash values that approximate character-level overlap. Since comparing every signature with every other signature is impossible at billions of documents, the signature is split into bands and hashed again. Documents that collide in a band become candidate duplicates. The system then finds connected components, so documents linked through different matches can be grouped together before retaining one document for training.
GPU processing changes the scale of fuzzy deduplication
Ryan's team initially ran fuzzy deduplication on a multi-node CPU setup and estimated that processing their planned data would take weeks. The GPU implementation runs the MinHash and locality-sensitive hashing computations across multiple nodes with Dask-cuDF. cuGraph computes connected components at the end. The full pipeline can run on GPUs, reducing the processing time substantially. Ryan also says ablation results showed a downstream model benefit compared with leaving the data undeduplicated.
Semantic deduplication compares meaning across modalities
Semantic deduplication embeds each dataset element, whether it is text, an image, or another modality, and compares the resulting vectors. A full pairwise comparison is too expensive, so the embeddings are first grouped with K-means clustering. Similarity is then computed within each cluster. When several embeddings are close enough, the item nearest the cluster centroid is retained and the others are removed. Ryan says this method was especially valuable for image curation and could match a baseline model's performance with half as many training iterations.
Embedding and clustering work can also benefit from GPU optimization
Most of the semantic deduplication time goes into creating embeddings. Ryan's team moved K-means clustering from the CPU to the GPU with cuML and also optimized embedding creation. He describes the main result as the ability to scale the open-source solution across multiple nodes and GPUs. The same classifier-oriented techniques later discussed, including GPU tokenization, can also speed up embedding creation without changing the underlying model.
Classifier inference can improve through batching and tokenization changes
The final pipeline stages may use classifiers with 100 million to 1 billion parameters for domain labels or quality scores. These models already run on GPUs, but Ryan says they can still be optimized without modifying the model. By profiling memory use at different sequence lengths and batch sizes, the system can increase the batch size for batches containing shorter sequences. Some models also benefit from GPU tokenization. In Ryan's test, these techniques improved inference by 40% using plain PyTorch and the existing model.
The right definition of a duplicate depends on the training purpose
During questions, Ryan agrees that deduplication is business dependent. Two documents with the same legal boilerplate but different names might be duplicates for foundation-model pretraining, while they could remain separate if the goal is to find document templates. For pretraining, his focus is on avoiding repeated tokens and getting fresher data for the model. The choice of thresholds and bucket counts therefore depends on what the dataset is meant to support and how much approximation it can tolerate.
