# Large Model Training and Inference with DeepSpeed

Samyam Rajbhandari, Microsoft DeepSpeed | LLMs in Production 2023 | 36:23

Source: https://www.youtube.com/watch?v=cntxC3g22oU
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/large-model-training-and-inference-with-deepspeed
Published: 2023-06-29
Tags: deployment, gpus, inference, latency

## TL;DR
- DeepSpeed used system techniques to increase model scale far faster than GPU memory capacity increased.
- ZeRO reduces memory use by partitioning model states across GPUs, while 3D parallelism combines pipeline, model, and ZeRO parallelism for very large models.
- DeepSpeed-Inference and DeepSpeed-MII apply different optimizations for latency and throughput, so large models can be deployed with less manual configuration.

## Summary
Samyam Rajbhandari explains how DeepSpeed developed from an early fix for slow distributed training into a collection of systems for large-model training and inference. The first breakthrough was delaying gradient synchronization until after several accumulation steps, which reduced communication and made 64-GPU training practical. ZeRO then avoided storing redundant optimizer, model, and gradient states on every GPU. 3D parallelism combined pipeline, model, and ZeRO parallelism to scale to trillion-parameter experiments and support models such as Megatron-Turing 530B and BLOOM 176B. Rajbhandari also describes mixture-of-experts models, which activate only some parameters for each token, and ZeRO-Infinity, which uses CPU memory and NVMe alongside GPU memory. For deployment, DeepSpeed-Inference and DeepSpeed-MII target different latency and throughput needs. He is candid about the remaining operational problem: fault tolerance for very large clusters is still not fully built into DeepSpeed.

## Key ideas
### DeepSpeed grew from a communication problem in distributed training
[01:59](https://www.youtube.com/watch?v=cntxC3g22oU&t=119s)
Rajbhandari describes training BERT on 64 V100 GPUs connected by a slow four-gigabit-per-second Ethernet network. Training on the cluster was slower than training on one GPU because every gradient accumulation step synchronized gradients across all devices. The team moved that synchronization to the end of the accumulation window. This reduced communication time by 16 times and made the 64-GPU run faster than the single-GPU baseline. That work became the starting point for DeepSpeed.

### ZeRO removes redundant copies of model state
[09:30](https://www.youtube.com/watch?v=cntxC3g22oU&t=570s)
Traditional data parallelism replicated the model and limited training to the memory of one GPU. Model parallelism partitioned the model but required large amounts of activation communication, which made scaling beyond a single node difficult. Rajbhandari's team instead partitioned optimizer states, model states, and gradients across GPUs, bringing only the required data to each device. This became ZeRO, or Zero Redundancy Optimizer. It allowed DeepSpeed to train models with hundreds of billions or even trillions of parameters without the communication cost associated with conventional model parallelism.

### 3D parallelism combines complementary ways to scale
[13:33](https://www.youtube.com/watch?v=cntxC3g22oU&t=813s)
When the team considered trillion-parameter training, it combined pipeline parallelism, model parallelism, and ZeRO. Pipeline parallelism was useful across nodes because it created relatively little communication overhead. The system was difficult to develop and use, but a test on more than a thousand GPUs produced a trillion-parameter run and near-linear scaling on 800 V100 GPUs. The same approach supported Megatron-Turing 530B, trained with 2,000 A100 GPUs, and BLOOM 176B.

### Dense models become impractical when training data grows
[16:42](https://www.youtube.com/watch?v=cntxC3g22oU&t=1002s)
Rajbhandari argues that simply making dense models larger creates an untenable training bill. Megatron-Turing 530B took about two months on 2,000 GPUs while using fewer than 300 billion tokens. Training a similarly large model with a trillion tokens could take six months to a year on that hardware, and using ten trillion tokens could take ten years. Mixture-of-experts models address this by routing each token to only a subset of the parameters. In his example, a 1.3-billion-parameter model with 128 experts reached accuracy similar to a 6.7-billion-parameter dense model while running about five times faster.

### ZeRO-Infinity uses distributed slow memory to fit larger models
[19:16](https://www.youtube.com/watch?v=cntxC3g22oU&t=1156s)
GPU memory is much smaller than the combined CPU memory and NVMe storage available in a system, but moving data from those slower locations through PCIe creates a bottleneck. The idea behind ZeRO-Infinity was to partition parameters across GPUs and move data from slower memory in parallel. The aggregate bandwidth from many concurrent transfers can reach hundreds of gigabytes per second, rather than the bandwidth of one transfer. This lets users fine-tune models with hundreds of billions of parameters on a single GPU when sufficient NVMe storage is available.

### Inference needs different optimization choices for different workloads
[22:22](https://www.youtube.com/watch?v=cntxC3g22oU&t=1342s)
DeepSpeed-Inference targets a broad range of model sizes, dense and sparse architectures, and deployment conditions. Offline workloads may prioritize throughput and cost, while customer-facing workloads may prioritize latency. Rajbhandari says DeepSpeed composes several optimizations to suit those different goals. DeepSpeed-MII packages optimized implementations of popular open-source models so users can deploy a Hugging Face model with a small amount of configuration instead of selecting each inference optimization themselves.

### DeepSpeed focuses on making large-model systems usable
[26:13](https://www.youtube.com/watch?v=cntxC3g22oU&t=1573s)
Rajbhandari says ZeRO can scale a model from 1.4 billion parameters to trillions with little or no application-code change. DeepSpeed can be enabled through configuration and launch settings, and it has integrations with Hugging Face and PyTorch. The system also runs on Azure virtual machines, other cloud infrastructure, and user-provided hardware. The main interface is a DeepSpeed engine wrapped around the model plus a configuration file containing the selected optimizations.

### Large clusters still make fault tolerance difficult
[32:43](https://www.youtube.com/watch?v=cntxC3g22oU&t=1963s)
In the question period, Rajbhandari says elasticity and fault tolerance have been discussed but are not fully built into DeepSpeed. Node failures and hardware errors become regular problems when training across tens, hundreds, or thousands of machines. His practical suggestion is to run an all-reduce bandwidth test before training, then use a binary search to locate the faulty part of a cluster when the result is wrong. This is a useful operational workaround, but he does not present it as a complete automation of failure handling.

## Notable quotes
- "The model scale, speed and democratization are kind of the three main emphasis points for DeepSpeed." (06:11)
- "We don't redundantly store any optimizer model states across GPUs." (11:16)
- "The fundamental idea behind ZeRO-Infinity, we build upon it and we release ZeRO-Infinity as part of DeepSpeed." (22:07)
- "All of these features do not mean much if they are very hard to use." (27:00)
- "It is definitely a pain point, especially when you're scaling to tens, hundreds or thousands of GPUs." (32:45)

## Tools & references mentioned
- DeepSpeed
- Microsoft
- ZeRO
- ZeRO-Infinity
- DeepSpeed-Inference
- DeepSpeed-MII
- DeepSpeed Chat
- BERT
- NVIDIA Apex
- PyTorch
- Hugging Face
- Microsoft Turing-NLG 17.2B
- BLOOM 176B
- Megatron-Turing 530B
- GPT-2
- OpenAI
- Megatron
- LLaMA 65B
- MPT 7B
- mixture-of-experts models
- Stable Diffusion
- PyTorch Fully Sharded Data Parallelism
- Colossal-AI
- Gemini

## Who should watch
- You are deciding how to train or fine-tune a model that does not fit in one GPU's memory.
- Your distributed training job spends too much time synchronizing gradients or moving model data between devices.
- You need to choose between latency-focused and throughput-focused inference for a large model.

## Related talks

- [Enabling Efficient Trillion Parameter Scale Training for Deep Learning Models](https://mlopstalks.com/talks/enabling-efficient-trillion-parameter-scale-training-for-deep-learning-models) (Tunji Ruwase, Microsoft, 27:36)
- [Quantized LLM Training at Scale with ZeRO++](https://mlopstalks.com/talks/quantized-llm-training-at-scale-with-zero) (Guanhua Wang, Microsoft, 24:35)
- [How to Optimize Large AI Models with PyTorch](https://mlopstalks.com/talks/how-to-optimize-large-ai-models-with-pytorch) (Michael Gschwind, Meta Platforms, 57:44)
- [Scaling Real-time Machine Learning at Chime](https://mlopstalks.com/talks/scaling-real-time-machine-learning-at-chime) (Peeyush Agarwal, Chime, 24:22)
- [AWS Trainium and Inferentia](https://mlopstalks.com/talks/aws-trainium-and-inferentia) (Kamran Khan, Annapurna ML & Matthew McClean, AWS, Annapurna Labs, 45:23)