# Quantized LLM Training at Scale with ZeRO++

Guanhua Wang, Microsoft | AI in Production 2025 | 24:35

Source: https://www.youtube.com/watch?v=G2tJe-uYfGQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/quantized-llm-training-at-scale-with-zero
Published: 2025-09-29
Tags: gpus, inference

## TL;DR
- ZeRO++ reduces communication in ZeRO Stage 3 training from 3M to 0.75M by quantizing weights and gradients and avoiding cross-node weight gathering.
- Its QGZ gradient protocol uses hierarchical quantization, two-hop communication, tensor-slice reordering, kernel fusion, and communication overlap.
- Guanhua Wang reports speedups of up to 2.16x for large language model training, 3.3x for Llama 70B RLHF training, and over 50% shorter training time in a LinkedIn production workload.

## Summary
Guanhua Wang explains that communication becomes a major cost when large language models use many GPUs, especially with small micro-batches or limited inter-node bandwidth. ZeRO Stage 3 uses three collective operations per training iteration, with a total communication volume of 3M for a model of size M. ZeRO++ reduces this to 0.75M. QWZ uses blockwise 8-bit weight quantization, HPZ keeps a secondary model replica within each machine, and QGZ replaces ring-based gradient reduce-scatter with a hierarchical quantized protocol. QGZ addresses sequential quantization overhead, communication-volume growth, and incorrect data placement. The implementation also fuses kernels and overlaps communication across data chunks. Wang reports up to 2.16x speedup over ZeRO for large language model training, 3.3x for Llama 70B RLHF training, and over 50% training-time reduction in a LinkedIn production workload. Eight-bit quantization usually preserves usable models, although very large models can show some accuracy loss.

## Key ideas
### Small micro-batches make communication a larger part of training time
[01:21](https://www.youtube.com/watch?v=G2tJe-uYfGQ&t=81s)
As models grow, training uses more GPUs while the maximum global batch size for a specific model stays fixed. Each GPU therefore receives a smaller micro-batch. Wang shows communication rising from 26% of training time at a micro-batch size of 24 to 44% at a size of 8. Limited inter-node bandwidth creates another bottleneck. With fewer InfiniBand links between nodes, the achieved FLOPS per GPU can fall to about half of the result with eight links.

### ZeRO Stage 3 communicates three model-sized payloads per iteration
[03:06](https://www.youtube.com/watch?v=G2tJe-uYfGQ&t=186s)
In ZeRO Stage 3, GPUs gather the full weights before forward computation, gather them again before backward computation, and then perform a reduce-scatter on gradients. Wang models each collective as communicating M values for a model of size M. The three operations therefore communicate 3M data per training iteration. ZeRO++ targets each operation separately rather than applying one quantization method everywhere.

### Blockwise weight quantization cuts forward-gather traffic in half
[05:27](https://www.youtube.com/watch?v=G2tJe-uYfGQ&t=327s)
QWZ reduces the forward all-gather payload by communicating 8-bit values instead of traditional 16-bit values. Wang says naive global quantization can introduce enough error to cause model divergence. QWZ divides the weights into smaller blocks and quantizes within each block. He reports a 3.3x improvement in data precision over the baseline and a quantization kernel that runs 2.5x faster than PyTorch alternatives. The forward-gather volume falls from M to 0.5M.

### HPZ trades extra memory for less cross-node weight communication
[07:01](https://www.youtube.com/watch?v=G2tJe-uYfGQ&t=421s)
Vanilla ZeRO Stage 3 partitions the full model across all devices, so backward all-gather communication crosses machines. HPZ, or heterogeneous partitioning in ZeRO, keeps a secondary model replica within each machine. Backward all-gather then happens within the machine rather than across machines. Wang describes this as a memory tradeoff. With the replica in place, cross-node backward all-gather volume falls from M to zero.

### QGZ keeps reductions in full precision while sending quantized gradients
[08:14](https://www.youtube.com/watch?v=G2tJe-uYfGQ&t=494s)
Directly quantizing gradients during reduce-scatter causes precision loss because the operation sums values from multiple GPUs. QGZ sends gradients in 4 or 8 bits while performing the reductions in full data precision. Its protocol replaces the ring-based reduce-scatter, where every hop performs dequantization, addition, and requantization. The replacement reduces the number of sequential quantization and dequantization steps from the number of GPUs to one.

### Two-hop communication prevents QGZ traffic from multiplying across GPUs
[12:52](https://www.youtube.com/watch?v=G2tJe-uYfGQ&t=772s)
A one-hop all-to-all protocol would reduce local gradients on each GPU but could increase cross-node traffic to N times M/4 when there are N GPUs per node. QGZ instead performs an intra-node quantized communication and reduction first. Each GPU then sends only M/(4N) data across nodes, followed by an inter-node operation. This two-hop design avoids the communication-volume blow-up of the simpler one-hop approach.

### Tensor-slice reordering fixes incorrect gradient placement
[15:04](https://www.youtube.com/watch?v=G2tJe-uYfGQ&t=904s)
Hierarchical communication can leave gradient chunks on the wrong GPUs after the intra-node and inter-node reductions. QGZ solves this with tensor-slice reordering before quantization and communication. GPUs first swap the order of local gradient chunks, including reversing selected chunks in the example. After the two reduction stages, each GPU receives the gradient chunk assigned to it.

### The implementation combines kernel fusion with communication overlap
[17:19](https://www.youtube.com/watch?v=G2tJe-uYfGQ&t=1039s)
QGZ fuses tensor-slice reordering with intra-node quantization, then fuses intra-node dequantization and reduction with inter-node quantization. This reduces global-memory input and output operations on the GPUs. It also overlaps intra-node and inter-node communication on independent data chunks. Wang presents both changes as system optimizations for lowering the protocol's latency.

### Reported gains depend on model size, workload, and network bandwidth
[18:32](https://www.youtube.com/watch?v=G2tJe-uYfGQ&t=1112s)
For small-batch training over a 100 Gbps inter-node connection, Wang reports up to 2.16x speedup over ZeRO for large language model training. With eight InfiniBand connections, he reports roughly 16% to 30% speedup. The system scales from 64 GPUs to more than 300 GPUs and shows its strongest gains with lower cross-node bandwidth. Validation-loss curves for GPT models with 350 million and 13 billion parameters match ZeRO Stage 3. For RLHF training, Wang reports 3.3x speedup on Llama 70B and 3x on OPT 30B.

## Notable quotes
- "Communication is the major bottleneck in large-scale LLM training." (00:30)
- "In total for every training iteration we need to communicate the 3M data volume given the model size of M." (05:04)
- "By doing this two-hop instead of one-hop we can remove this communication volume blow-up issue." (14:14)
- "For Llama 70B we can achieve 3.3x speedup and for OPT 30B we achieve 3x speedup." (20:44)
- "If we quantize to 8 bits it is okay, or the fully trained model is usable." (23:28)

## Tools & references mentioned
- ZeRO++
- DeepSpeed
- Microsoft
- LinkedIn
- QWZ
- HPZ
- QGZ
- PyTorch
- Megatron-Turing NLG
- GPT
- Llama 70B
- OPT 30B
- Phi-3
- InfiniBand

## Who should watch
- You are training large language models with ZeRO Stage 3 and communication time is limiting GPU utilization.
- Your cluster has modest inter-node bandwidth and you want to understand where quantized collectives can reduce training time.
- You need practical details about the memory, precision, data-placement, and convergence tradeoffs of quantized training.

## Related talks

- [Enabling Efficient Trillion Parameter Scale Training for Deep Learning Models](https://mlopstalks.com/talks/enabling-efficient-trillion-parameter-scale-training-for-deep-learning-models) (Tunji Ruwase, Microsoft, 27:36)
- [Large Model Training and Inference with DeepSpeed](https://mlopstalks.com/talks/large-model-training-and-inference-with-deepspeed) (Samyam Rajbhandari, Microsoft DeepSpeed, 36:23)
- [LLM Distillation and Compression](https://mlopstalks.com/talks/llm-distillation-and-compression) (Guanhua "Alex" Wang, Microsoft, 49:48)
- [Making LLM Inference Affordable](https://mlopstalks.com/talks/making-llm-inference-affordable) (Daniel Campos, Snowflake, 32:07)
- [Efficiently Scaling and Deploying LLMs](https://mlopstalks.com/talks/efficiently-scaling-and-deploying-llms) (Hanlin Tang, MosaicML, 25:14)