# Building RedPajama

Vipul Ved Prakash, Together | LLMs in Production 2023 | 27:52

Source: https://www.youtube.com/watch?v=4EQh9cbPW7M
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-redpajama
Published: 2023-08-21
Tags: data-quality, inference, open-models, training-pipelines

## TL;DR
- Progress in AI depends mainly on growing data and compute, which creates pressure on data quality, model size, and infrastructure.
- RedPajama aims to make open models reproducible by releasing open weights, licensing, data, and the recipe used to create the data.
- Open models can be adapted with few-shot prompting, instruction tuning, continued pre-training, and local deployment, but evaluation and data quality still need substantial work.

## Summary
Vipul Ved Prakash explains why building capable language models requires more than publishing model weights. Data volume, compute, model size, and infrastructure all scale together, while data quality becomes harder and more expensive to manage. RedPajama responds by opening the weights, license, training data, and data-generation recipe. Together used a trillion-token dataset, the Pythia architecture, open-source tooling, and public infrastructure to create 3B and 7B models. Vipul discusses their benchmark results, quantization, CPU and iPhone use, and ways to adapt the models with prompting or private instruction data. He is also direct about unresolved problems. Benchmark results can be hard to reproduce, training thousands of GPUs wastes substantial capacity, and data mixtures, deduplication, filtering, and contamination require more study. The talk makes a practical case for open models while acknowledging that openness does not remove the engineering cost of training and evaluating them.

## Key ideas
### Data and compute are the main forces behind better AI models
[02:08](https://www.youtube.com/watch?v=4EQh9cbPW7M&t=128s)
Vipul reduces recent AI progress to two main drivers: data and compute. Scaling laws show that larger datasets generally produce better models, while training those models requires increasing amounts of hardware. He compares ImageNet's three million images from a decade ago with image models trained on three billion images or more today. He also describes GPT-3 training as requiring 1,800 GPUs running continuously for a month. More data creates pressure on data cleaning and acquisition, while larger models require more memory and floating-point operations for training and inference.

### RedPajama treats openness as more than publishing model weights
[06:39](https://www.youtube.com/watch?v=4EQh9cbPW7M&t=399s)
RedPajama is designed around four kinds of openness: weights, licenses, data, and the recipe that creates the data. Open weights allow fine-tuning, private deployment, and use on devices. An open license allows commercial applications. Open data gives users a way to inspect, refine, filter, and remove problematic material. An open recipe makes it possible to reproduce the model from scratch or create a version for another language. Vipul argues that all four are needed for community-driven improvement.

### The RedPajama training process follows an open and reproducible stack
[08:17](https://www.youtube.com/watch?v=4EQh9cbPW7M&t=497s)
For its dataset, RedPajama followed the seven data slices and filtering process described in the Llama paper, then released the associated code. The project used roughly a trillion tokens and the Pythia Transformer architecture. It also instruction-tuned and chat-tuned the models. The workflow used Slurm and Spark for data processing, DeepSpeed for pre-training, and the OpenChatGPT system for instruction and chat tuning. The models were trained on the Summit supercomputer at Oak Ridge National Laboratory with 3,300 GPUs, and the data, checkpoints, and intermediate checkpoints were published on Hugging Face under Apache 2.

### Small RedPajama models can be useful after instruction tuning and quantization
[10:26](https://www.youtube.com/watch?v=4EQh9cbPW7M&t=626s)
Vipul says the first RedPajama run produced 3B and 7B models. The 3B base model was six points behind Llama 7B on the comparison he presented, while the 3B instruction model was one point behind. The models could be quantized and brought to CPUs and iPhones through work with llama.cpp and MLC. The 7B model also improved on GPT-J, and instruction tuning made it useful for few-shot sentiment analysis, classification, and data extraction. Vipul presents these models as practical alternatives when applications need open weights.

### Open data enables contamination checks and more honest evaluation
[12:32](https://www.youtube.com/watch?v=4EQh9cbPW7M&t=752s)
RedPajama's open data makes it possible to inspect whether benchmark examples appear in the instruction data. Vipul focuses on contamination because it can produce an inflated impression of model quality. He says users applying the model to downstream tasks should be able to verify that the data has been decontaminated. He also questions the reliability of current evaluation practices. A technical report attempting to reproduce MMLU obtained substantially different results from the commonly cited reference, which makes it harder to use benchmarks as dependable guides for model development.

### Data mixtures, weighting, and deduplication remain open research problems
[15:03](https://www.youtube.com/watch?v=4EQh9cbPW7M&t=903s)
Vipul says RedPajama used the mixture described in the Llama paper, but different weights for the data slices can produce a better final model. Large sources such as Common Crawl may also need to be divided into smaller parts and scored separately. SlimPajama, which he mentions as a more aggressively deduplicated dataset, is about half the size of the original dataset. The community still needs to determine the right level of deduplication and how it interacts with data mixtures. Filtering, bias reduction, and detoxification are additional parts of this work.

### Large-scale training wastes capacity when GPU utilization falls
[16:43](https://www.youtube.com/watch?v=4EQh9cbPW7M&t=1003s)
Scaling training across thousands of GPUs creates a utilization problem. Vipul compares an ideal Summit run with the actual RedPajama run and says the results were fairly close, but at roughly 3,000 GPUs the difference corresponded to about 500 idle GPUs. He estimates that this gap can represent millions of dollars in computing resources. Together is working on communication optimization and on techniques that can be used across different training tasks without custom work for every model. He mentions Cocktail SGD as an approach for low-network distributed settings and high-performance computing environments.

### Open models can be adapted through several levels of customization
[19:12](https://www.youtube.com/watch?v=4EQh9cbPW7M&t=1152s)
Vipul recommends starting with few-shot prompting, where one or two examples accompany the task prompt. Users can then create instruction data and tune a base model for their own application. With enough data, they can continue pre-training and mix the original open dataset with private data to reduce catastrophic forgetting. Quantization provides another way to fit the model to a deployment environment. He also describes Together's AI cloud service, which offers inference, hosting, fine-tuning, and training, while noting that the same models can run in a private VPC, on-premises, or on devices.

## Notable quotes
- "The two driving forces behind progress in AI can be boiled down to data and compute." (02:08)
- "We want open weights, open license, open data, and the recipe that creates the data." (07:03)
- "We need to find better ways of doing these evaluations, making sure that they're reproducible and unbiased." (14:37)
- "A big part of monotonic progress is evaluation. We need to know how well these models are performing." (13:43)
- "You can get an incredible amount of customization with a few thousand examples." (25:14)

## Tools & references mentioned
- Together
- RedPajama
- Stable Diffusion
- OpenAI
- GPT-3
- Llama
- Pythia
- Slurm
- Spark
- DeepSpeed
- OpenChatGPT
- HELM
- LM Harness
- EleutherAI
- Oak Ridge National Laboratory
- Hugging Face
- Apache 2
- llama.cpp
- MLC
- GPT-J
- MPT
- Falcon
- MMLU
- SlimPajama
- Cocktail SGD
- ICML
- Together AI Cloud

## Who should watch
- You are deciding whether open model weights are worth the work of private deployment and customization.
- You need to understand what must be released to make an LLM genuinely reproducible, rather than merely downloadable.
- You are training models at scale and want concrete discussion of data mixtures, benchmark contamination, GPU utilization, or continued pre-training.

## Editor's note

Vipul Ved Prakash argues that open models need the training data and recipe, not only published weights, so others can reproduce the work. ZenML records each pipeline run's steps, inputs, outputs, and code version, allowing a model or artifact to be traced to its source. Its stack configuration also lets the same pipeline code run across different infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLOps + Machine Learning](https://mlopstalks.com/talks/mlops-machine-learning) (James Sutton, Algorithmia, 1:01:50)
- [Enterprises Using MLOps, the Changing LLM Landscape, MLOps Pipelines](https://mlopstalks.com/talks/enterprises-using-mlops-the-changing-llm-landscape-mlops-pipelines) (Chris Van Pelt, Weights & Biases, 47:51)
- [A Playground for AI Engineers](https://mlopstalks.com/talks/a-playground-for-ai-engineers) (Paulo Vasconcellos, Hotmart, 54:42)
- [Productionizing AI: How to Think From the End](https://mlopstalks.com/talks/productionizing-ai-how-to-think-from-the-end) (Annie Condon, 11:11)
- [Considerations and Optimizations for Deploying Open Source LLMs at Your Company](https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company) (Oscar Rovira, Mystic AI, 11:31)