DeepSpeed used system techniques to increase model scale far faster than GPU memory capacity increased.
2
ZeRO reduces memory use by partitioning model states across GPUs, while 3D parallelism combines pipeline, model, and ZeRO parallelism for very large models.
3
DeepSpeed-Inference and DeepSpeed-MII apply different optimizations for latency and throughput, so large models can be deployed with less manual configuration.
Summary
Samyam Rajbhandari explains how DeepSpeed developed from an early fix for slow distributed training into a collection of systems for large-model training and inference. The first breakthrough was delaying gradient synchronization until after several accumulation steps, which reduced communication and made 64-GPU training practical. ZeRO then avoided storing redundant optimizer, model, and gradient states on every GPU. 3D parallelism combined pipeline, model, and ZeRO parallelism to scale to trillion-parameter experiments and support models such as Megatron-Turing 530B and BLOOM 176B. Rajbhandari also describes mixture-of-experts models, which activate only some parameters for each token, and ZeRO-Infinity, which uses CPU memory and NVMe alongside GPU memory. For deployment, DeepSpeed-Inference and DeepSpeed-MII target different latency and throughput needs. He is candid about the remaining operational problem: fault tolerance for very large clusters is still not fully built into DeepSpeed.
DeepSpeed grew from a communication problem in distributed training
Rajbhandari describes training BERT on 64 V100 GPUs connected by a slow four-gigabit-per-second Ethernet network. Training on the cluster was slower than training on one GPU because every gradient accumulation step synchronized gradients across all devices. The team moved that synchronization to the end of the accumulation window. This reduced communication time by 16 times and made the 64-GPU run faster than the single-GPU baseline. That work became the starting point for DeepSpeed.
Traditional data parallelism replicated the model and limited training to the memory of one GPU. Model parallelism partitioned the model but required large amounts of activation communication, which made scaling beyond a single node difficult. Rajbhandari's team instead partitioned optimizer states, model states, and gradients across GPUs, bringing only the required data to each device. This became ZeRO, or Zero Redundancy Optimizer. It allowed DeepSpeed to train models with hundreds of billions or even trillions of parameters without the communication cost associated with conventional model parallelism.
3D parallelism combines complementary ways to scale
When the team considered trillion-parameter training, it combined pipeline parallelism, model parallelism, and ZeRO. Pipeline parallelism was useful across nodes because it created relatively little communication overhead. The system was difficult to develop and use, but a test on more than a thousand GPUs produced a trillion-parameter run and near-linear scaling on 800 V100 GPUs. The same approach supported Megatron-Turing 530B, trained with 2,000 A100 GPUs, and BLOOM 176B.
Dense models become impractical when training data grows
Rajbhandari argues that simply making dense models larger creates an untenable training bill. Megatron-Turing 530B took about two months on 2,000 GPUs while using fewer than 300 billion tokens. Training a similarly large model with a trillion tokens could take six months to a year on that hardware, and using ten trillion tokens could take ten years. Mixture-of-experts models address this by routing each token to only a subset of the parameters. In his example, a 1.3-billion-parameter model with 128 experts reached accuracy similar to a 6.7-billion-parameter dense model while running about five times faster.
ZeRO-Infinity uses distributed slow memory to fit larger models
GPU memory is much smaller than the combined CPU memory and NVMe storage available in a system, but moving data from those slower locations through PCIe creates a bottleneck. The idea behind ZeRO-Infinity was to partition parameters across GPUs and move data from slower memory in parallel. The aggregate bandwidth from many concurrent transfers can reach hundreds of gigabytes per second, rather than the bandwidth of one transfer. This lets users fine-tune models with hundreds of billions of parameters on a single GPU when sufficient NVMe storage is available.
Inference needs different optimization choices for different workloads
DeepSpeed-Inference targets a broad range of model sizes, dense and sparse architectures, and deployment conditions. Offline workloads may prioritize throughput and cost, while customer-facing workloads may prioritize latency. Rajbhandari says DeepSpeed composes several optimizations to suit those different goals. DeepSpeed-MII packages optimized implementations of popular open-source models so users can deploy a Hugging Face model with a small amount of configuration instead of selecting each inference optimization themselves.
DeepSpeed focuses on making large-model systems usable
Rajbhandari says ZeRO can scale a model from 1.4 billion parameters to trillions with little or no application-code change. DeepSpeed can be enabled through configuration and launch settings, and it has integrations with Hugging Face and PyTorch. The system also runs on Azure virtual machines, other cloud infrastructure, and user-provided hardware. The main interface is a DeepSpeed engine wrapped around the model plus a configuration file containing the selected optimizations.
Large clusters still make fault tolerance difficult
In the question period, Rajbhandari says elasticity and fault tolerance have been discussed but are not fully built into DeepSpeed. Node failures and hardware errors become regular problems when training across tens, hundreds, or thousands of machines. His practical suggestion is to run an all-reduce bandwidth test before training, then use a binary search to locate the faulty part of a cluster when the result is wrong. This is a useful operational workaround, but he does not present it as a complete automation of failure handling.
