Adding GPUs made training more expensive, but barely shortened the run. The accelerator may be waiting for data, moving model state, synchronizing gradients or recovering progress lost to failures. Each bottleneck needs a different experiment. Start with a training job whose utilization improves when its inputs are already in memory, then examine how a data engine prepares and loads those inputs. Move inside the GPU to understand memory movement before partitioning model state across devices. Distributed-training accounts then separate useful parallelism from communication that grows faster than the work. Quantized collectives add another trade-off between memory, bandwidth and precision. Finish with checkpoint recovery and shared-cluster placement, so a faster iteration also becomes a training run that finishes reliably. Reported gains belong to the speakers' workloads.
Why first: First isolate whether the model can keep a GPU busy at all. Mittal loads a training slice into RAM, then follows the normal path until cached raw data still leaves a conversion bottleneck. Caching transformed output solves a different problem from caching remote reads. That diagnosis tells you what to change before adding compute.
Why here: The loader begins with how the dataset is represented. Chia's 2024 demonstration keeps compact image metadata in Parquet and uses one interface to filter, inspect and feed training data, with local or distributed execution. It extends the first diagnosis upstream: prepare a usable representation and an efficient loading path together.
Why here: Once data reaches the accelerator, moving it through memory can still dominate the work. Fregly connects arithmetic intensity to hardware execution and explains why kernel operations overlap computation with transfers. This supplies the physical reason that model size, memory placement and parallel execution cannot be optimized independently.
Why here: Training memory includes gradients and optimizer state as well as weights. Ruwase explains ZeRO's successive partitioning stages and the option of moving state to CPU memory or NVMe. These mechanisms address fitting the training job; his discussion also makes clear that making a run possible is not the same as making it faster.
Why here: More devices only help when synchronization is proportionate to useful work. Rajbhandari's 2023 account begins with a cluster slower than one GPU, fixed by delaying gradient synchronization until the accumulation window ends. He then combines complementary forms of parallelism. Read the progression as an answer to communication overhead, not a target GPU count.
Why here: Partitioning can move the bottleneck onto the network, especially with small micro-batches. Wang's ZeRO++ account reduces collective traffic through quantization and local replicas, trading additional memory and precision decisions for less cross-node communication. The reported speedups vary with bandwidth and workload, so the useful next step is measuring those same constraints in your run.
Why here: A fast training step still wastes time if checkpoint writes pause every worker. Singh describes LinkedIn first saving in memory and streaming to storage asynchronously, alongside kernel fusion work. This joins recovery overhead to the earlier compute analysis while keeping proposed distributed restore caches separate from the changes already described as implemented.
Why last: Adobe makes completed jobs the final unit of success. Khanna describes recovering unhealthy hardware and resuming long runs, while placing each job within one cluster and preferably one zone for connectivity. The fleet can span clusters without forcing an individual training run across slow links. End by protecting useful progress as well as accelerating it.