Guanhua "Alex" Wang says high-quality, carefully cleaned data is the main reason Phi-3 achieved strong results despite its small size.
2
Wang says distillation often reduces accuracy on general-purpose models, while quantization is a better fit for compressing small models.
3
Domino overlaps GPU communication with computation by splitting data dependencies into smaller pieces, reaching up to 1.3x the throughput of Megatron on DGX-H100 systems.
Summary
Guanhua "Alex" Wang discusses how Microsoft trained and compressed small language models, drawing on his work with Phi-3 and DeepSpeed. He says Phi-3 depended on highly cleaned data, including purchased material from sources such as The New York Times and Forbes. Post-training matters especially for small models because it adapts them to user data and tasks. Wang describes DeepSpeed's ZeRO optimizer, CPU and NVMe offloading, quantization, and LoRA. He is direct about distillation: it reduced accuracy in the team's experiments, especially for general-purpose models, although it may work for a simple personal assistant. The second half focuses on Domino, a DeepSpeed training system that breaks communication into smaller pieces and overlaps it with computation. This hides much of the communication delay without changing the hardware or kernels. Wang says Domino reached a 1.3x speedup over Megatron and can extend tensor parallel training across multiple nodes.
Wang says the most important reason Phi-3 worked well at a small size was the quality of its data. The team removed noise through preprocessing and cleaning before training. It also purchased text from sources including The New York Times and Forbes because those materials were considered high quality and less likely to contain incorrect information. Synthetic data was used, but Wang says its share was small because it often repeats patterns already present in human-written data. The preparation took substantial time and money, but the model could not simply be taken off the shelf and expected to perform well.
Small models need substantial post-training after pre-training
Wang says post-training is more important than pre-training for small language models inside Microsoft because the models must learn customized data. He describes three stages. Pre-training uses the original purchased data, mid-training adds support for more languages, and post-training adapts the model to customized user text. The team keeps post-training sequences relatively short so the work fits on a smaller GPU cluster. Longer sequences create larger activations in the forward pass and larger gradients in the backward pass. He gives 4K training sequences as an example that can support a 1K or 2K context window.
DeepSpeed reduces memory pressure by partitioning and offloading model state
Wang describes DeepSpeed as a PyTorch-based library that adds features such as the ZeRO optimizer. Under ZeRO, each GPU keeps only part of the model state instead of the full model, then gathers the pieces needed for a layer's computation and releases them afterward. DeepSpeed can also move data from GPU memory to CPU memory when it is not needed, then bring it back for computation. ZeRO-Infinity extends this idea to NVMe storage, which has more capacity than CPU memory. Wang says this can make it possible to train a large model on one GPU, although moving data to disk can make training extremely slow.
Quantization is part of post-training and has a practical accuracy limit
The team quantizes model weights during post-training and has tried four-bit and three-bit formats. Wang says three bits is close to the practical limit because lower precision reduces the information retained in the weights. Four-bit or three-bit models may have slightly lower or roughly equal quality to the base model, but two-bit and one-bit versions lose a large amount of accuracy. He describes keeping a quantized weight tensor alongside the original and passing gradients through the quantized version during training. The team has used GPTQ and is also researching how to choose rounding directions for better accuracy.
Distillation can fit a personal assistant but fails on broad services
Wang says the team stopped using distillation because it consistently reduced accuracy, including on models with several tens of billions of parameters. He makes an exception for simple, narrow tasks, such as a personal assistant or a Copilot used by one person. A general assistant serving many users needs more of the model's stored information, and distillation can leave too little capacity for different user attributes. For a small local assistant, Wang describes learning from simple signals such as websites visited, viewing times, and video-game activity. The training can happen locally on the laptop without uploading the data to a server.
LoRA works best when post-training data resembles pre-training data
Wang says LoRA is used for post-training on Microsoft and open models, but its results depend on how different the new data is from the pre-training data. When the data has similar attributes, LoRA performs well. When it introduces substantially different information, the approach can lose accuracy. Wang connects this problem to model size. LoRA works well on models such as Llama 45B or Llama 70B, while smaller models with only a few billion parameters have less redundancy for a low-rank adapter to use. For those small models, he prefers pure quantization.
Domino hides communication by pipelining smaller independent pieces
Wang describes Domino as a system that hides GPU communication behind computation during both pre-training and post-training. It does not remove communication. Instead, it breaks a layer's data dependency into smaller pieces, starts communicating each piece as soon as it is ready, and continues computing other pieces. In his example, five seconds of computation and three seconds of communication previously produced an eight-second iteration. With the overlap, the communication fits inside the five seconds of computation, reducing the iteration to about five seconds. Wang says Domino can hide around 70% to 100% of communication, depending on the workload.
Domino improves multi-node training without hardware-specific changes
Wang says Domino achieved a 1.3x speedup over Megatron in a setting with high-bandwidth interconnects, close to the throughput he considers optimal there. He also says the approach extends tensor-parallel training across multiple nodes, where Megatron's visible communication overhead grows. Domino operates at the tensor and algorithm level. It does not modify GPU kernels or depend on a particular chip, which lets the team run it on AMD GPUs and other systems. Quantizing weights, activations, or gradients before transfer can reduce communication further when bandwidth is lower. Wang says Domino is open source and merged into the DeepSpeed master branch.
