# How GPUs are Revolutionizing AI Data Management

 | MLOps Community | 30:02

Source: https://www.youtube.com/watch?v=Ld4xz0Yascw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-gpus-are-revolutionizing-ai-data-management
Published: 2024-10-09
Tags: data-engineering, gpus, open-source

## TL;DR
- Data curation improves model accuracy by removing harmful training data and selecting the best subset that fits the compute budget.
- GPU acceleration makes fuzzy deduplication, semantic deduplication, embedding creation, clustering, and classifier inference practical at very large data volumes.
- NeMo Curator provides open-source tools for distributed, GPU-accelerated data curation across text, images, and other modalities.

## Summary
Ryan explains why data curation is necessary when foundation models train on enormous datasets. A curation pipeline starts with raw sources, extracts usable text, applies simple cleaning and filters, then performs deduplication and classifier-based annotation. The expensive stages become difficult when datasets reach hundreds of terabytes or billions of documents. Ryan describes GPU implementations for fuzzy deduplication with MinHash, locality-sensitive hashing, and connected components. He also covers semantic deduplication, which compares embeddings after clustering them to avoid an impossible all-pairs comparison. Classifier inference can be improved by changing batch sizes according to sequence length and using GPU tokenization. The talk focuses on the engineering tradeoffs behind scaling these methods, including approximate comparisons and business-dependent definitions of duplication. Ryan presents NeMo Curator as the open-source home for the tools his team developed.

## Key ideas
### Curation improves models by removing harmful data and choosing better training examples
[03:04](https://www.youtube.com/watch?v=Ld4xz0Yascw&t=184s)
Ryan says the success of data curation is measured by downstream model accuracy. Curation helps in two ways: it removes data that is actively harmful during training, and it selects the best subset when there is more data than the compute budget allows. Repeating tokens is usually less useful than seeing new tokens, so deduplication can improve the data mix while reducing wasted training. The pipeline should therefore spend its most expensive processing on the smaller set of documents that survives earlier filters.

### A curation pipeline moves from cheap filters to expensive models
[03:38](https://www.youtube.com/watch?v=Ld4xz0Yascw&t=218s)
The pipeline begins with raw data from cloud storage, a workstation, or a cluster. For web data such as Common Crawl, it extracts plain text from HTML, fixes encoding problems, identifies languages, and applies preliminary quality filters. Simple heuristics can remove repetitive documents or text with excessive punctuation. Later stages perform exact or fuzzy deduplication, semantic deduplication, and annotation with larger classifiers. Ryan says this ordering matters because datasets can contain hundreds of terabytes, making it impractical to run heavy models over everything.

### Fuzzy deduplication uses MinHash and locality-sensitive hashing to avoid all-pairs comparison
[08:07](https://www.youtube.com/watch?v=Ld4xz0Yascw&t=487s)
Fuzzy deduplication catches documents that are almost the same, such as legal documents sharing boilerplate with a few names changed. Each document receives a MinHash signature containing hash values that approximate character-level overlap. Since comparing every signature with every other signature is impossible at billions of documents, the signature is split into bands and hashed again. Documents that collide in a band become candidate duplicates. The system then finds connected components, so documents linked through different matches can be grouped together before retaining one document for training.

### GPU processing changes the scale of fuzzy deduplication
[10:19](https://www.youtube.com/watch?v=Ld4xz0Yascw&t=619s)
Ryan's team initially ran fuzzy deduplication on a multi-node CPU setup and estimated that processing their planned data would take weeks. The GPU implementation runs the MinHash and locality-sensitive hashing computations across multiple nodes with Dask-cuDF. cuGraph computes connected components at the end. The full pipeline can run on GPUs, reducing the processing time substantially. Ryan also says ablation results showed a downstream model benefit compared with leaving the data undeduplicated.

### Semantic deduplication compares meaning across modalities
[11:33](https://www.youtube.com/watch?v=Ld4xz0Yascw&t=693s)
Semantic deduplication embeds each dataset element, whether it is text, an image, or another modality, and compares the resulting vectors. A full pairwise comparison is too expensive, so the embeddings are first grouped with K-means clustering. Similarity is then computed within each cluster. When several embeddings are close enough, the item nearest the cluster centroid is retained and the others are removed. Ryan says this method was especially valuable for image curation and could match a baseline model's performance with half as many training iterations.

### Embedding and clustering work can also benefit from GPU optimization
[13:24](https://www.youtube.com/watch?v=Ld4xz0Yascw&t=804s)
Most of the semantic deduplication time goes into creating embeddings. Ryan's team moved K-means clustering from the CPU to the GPU with cuML and also optimized embedding creation. He describes the main result as the ability to scale the open-source solution across multiple nodes and GPUs. The same classifier-oriented techniques later discussed, including GPU tokenization, can also speed up embedding creation without changing the underlying model.

### Classifier inference can improve through batching and tokenization changes
[14:15](https://www.youtube.com/watch?v=Ld4xz0Yascw&t=855s)
The final pipeline stages may use classifiers with 100 million to 1 billion parameters for domain labels or quality scores. These models already run on GPUs, but Ryan says they can still be optimized without modifying the model. By profiling memory use at different sequence lengths and batch sizes, the system can increase the batch size for batches containing shorter sequences. Some models also benefit from GPU tokenization. In Ryan's test, these techniques improved inference by 40% using plain PyTorch and the existing model.

### The right definition of a duplicate depends on the training purpose
[19:09](https://www.youtube.com/watch?v=Ld4xz0Yascw&t=1149s)
During questions, Ryan agrees that deduplication is business dependent. Two documents with the same legal boilerplate but different names might be duplicates for foundation-model pretraining, while they could remain separate if the goal is to find document templates. For pretraining, his focus is on avoiding repeated tokens and getting fresher data for the model. The choice of thresholds and bucket counts therefore depends on what the dataset is meant to support and how much approximation it can tolerate.

## Notable quotes
- "Data curation is all in service of the model." (03:04)
- "The entire end to end pipeline now can run entirely on the GPU." (10:59)
- "Semantic deduplication is unique in the fact that it can be applied across modalities." (06:03)
- "This is a very business dependent." (19:41)
- "We can accelerate the inference of the model by 40% without changing anything in the model." (16:03)

## Tools & references mentioned
- NVIDIA
- NeMo Curator
- Common Crawl
- Llama 3.1
- Dask-cuDF
- cuGraph
- cuML
- PyTorch
- MinHash
- locality-sensitive hashing
- K-means clustering
- cosine similarity
- L2 norm
- BERT
- Docker

## Who should watch
- You are building a foundation model dataset and need to reduce duplicate or low-quality content before training.
- Your deduplication job is too slow on CPU infrastructure, especially when the input reaches billions of documents or hundreds of terabytes.
- You need practical GPU optimization ideas for embedding generation, clustering, or offline classifier inference.

## Related talks

- [Accelerating Growth Through Optimizing GPU Usage](https://mlopstalks.com/talks/accelerating-growth-through-optimizing-gpu-usage) (Sahil Khanna, Adobe, 23:53)
- [Efficient GPU infrastructure at LinkedIn](https://mlopstalks.com/talks/efficient-gpu-infrastructure-at-linkedin) (Animesh Singh, LinkedIn, 59:14)
- [Data-Centric AI Means Centralizing Training Data](https://mlopstalks.com/talks/data-centric-ai-means-centralizing-training-data) (Alberto Rizzoli, V7, 49:01)
- [Innovative Gen AI Applications: Beyond Text](https://mlopstalks.com/talks/innovative-gen-ai-applications-beyond-text) (Diana C. Montañes Mondragon & Nick Schenone, QuantumBlack, 54:45)
- [Accelerating Multimodal AI](https://mlopstalks.com/talks/accelerating-multimodal-ai) (Ethan Rosenthal, Runway, 54:58)