# Scaling training and compute A pack of 8 sessions from the MLOps Community YouTube channel, in the order to watch them. 5h 37m of video. Page: https://mlopstalks.com/packs/scaling-training-and-compute Adding GPUs made training more expensive, but barely shortened the run. The accelerator may be waiting for data, moving model state, synchronizing gradients or recovering progress lost to failures. Each bottleneck needs a different experiment. Start with a training job whose utilization improves when its inputs are already in memory, then examine how a data engine prepares and loads those inputs. Move inside the GPU to understand memory movement before partitioning model state across devices. Distributed-training accounts then separate useful parallelism from communication that grows faster than the work. Quantized collectives add another trade-off between memory, bandwidth and precision. Finish with checkpoint recovery and shared-cluster placement, so a faster iteration also becomes a training run that finishes reliably. Reported gains belong to the speakers' workloads. ## This pack is for you if - Your training GPUs stay idle while CPU workers read, decode or transform the next batch. - A model or optimizer state no longer fits, and adding devices creates communication overhead. - Long distributed runs lose progress to checkpoints, failed nodes and contention for shared capacity. ## The talks, in order ### 1. Fixing GPU Starvation in Large-Scale Distributed Training Kashish Mittal, Uber | 52:49 | MLOps Podcast Video: https://www.youtube.com/watch?v=1WFffCGhm7U Summary: https://mlopstalks.com/talks/fixing-gpu-starvation-in-large-scale-distributed-training.md Why first: First isolate whether the model can keep a GPU busy at all. Mittal loads a training slice into RAM, then follows the normal path until cached raw data still leaves a conversion bottleneck. Caching transformed output solves a different problem from caching remote reads. That diagnosis tells you what to change before adding compute. ### 2. The Daft distributed Python data engine: multimodal data curation at any scale Jay Chia, eventual | 26:34 | DE4AI 2024 Video: https://www.youtube.com/watch?v=kwWTcarDNTA Summary: https://mlopstalks.com/talks/the-daft-distributed-python-data-engine-multimodal-data-curation-at-any-scale.md Why here: The loader begins with how the dataset is represented. Chia's 2024 demonstration keeps compact image metadata in Parquet and uses one interface to filter, inspect and feed training data, with local or distributed execution. It extends the first diagnosis upstream: prepare a usable representation and an efficient loading path together. ### 3. Performance Optimization and Software/Hardware Co-design across PyTorch, CUDA, and NVIDIA GPUs Chris Fregly, AI performance engineer, startup founder, and investor | 1:25:50 | MLOps Podcast Video: https://www.youtube.com/watch?v=KfgkK27n4ec Summary: https://mlopstalks.com/talks/performance-optimization-and-software-hardware-co-design-across-pytorch-cuda.md Why here: Once data reaches the accelerator, moving it through memory can still dominate the work. Fregly connects arithmetic intensity to hardware execution and explains why kernel operations overlap computation with transfers. This supplies the physical reason that model size, memory placement and parallel execution cannot be optimized independently. ### 4. Enabling Efficient Trillion Parameter Scale Training for Deep Learning Models Tunji Ruwase, Microsoft | 27:36 | AI in Production 2024 Video: https://www.youtube.com/watch?v=JFBACDiNRzk Summary: https://mlopstalks.com/talks/enabling-efficient-trillion-parameter-scale-training-for-deep-learning-models.md Why here: Training memory includes gradients and optimizer state as well as weights. Ruwase explains ZeRO's successive partitioning stages and the option of moving state to CPU memory or NVMe. These mechanisms address fitting the training job; his discussion also makes clear that making a run possible is not the same as making it faster. ### 5. Large Model Training and Inference with DeepSpeed Samyam Rajbhandari, Microsoft DeepSpeed | 36:23 | LLMs in Production 2023 Video: https://www.youtube.com/watch?v=cntxC3g22oU Summary: https://mlopstalks.com/talks/large-model-training-and-inference-with-deepspeed.md Why here: More devices only help when synchronization is proportionate to useful work. Rajbhandari's 2023 account begins with a cluster slower than one GPU, fixed by delaying gradient synchronization until the accumulation window ends. He then combines complementary forms of parallelism. Read the progression as an answer to communication overhead, not a target GPU count. ### 6. Quantized LLM Training at Scale with ZeRO++ Guanhua Wang, Microsoft | 24:35 | AI in Production 2025 Video: https://www.youtube.com/watch?v=G2tJe-uYfGQ Summary: https://mlopstalks.com/talks/quantized-llm-training-at-scale-with-zero.md Why here: Partitioning can move the bottleneck onto the network, especially with small micro-batches. Wang's ZeRO++ account reduces collective traffic through quantization and local replicas, trading additional memory and precision decisions for less cross-node communication. The reported speedups vary with bandwidth and workload, so the useful next step is measuring those same constraints in your run. ### 7. Efficient GPU infrastructure at LinkedIn Animesh Singh, LinkedIn | 59:14 | MLOps Podcast Video: https://www.youtube.com/watch?v=NvQPFakVe60 Summary: https://mlopstalks.com/talks/efficient-gpu-infrastructure-at-linkedin.md Why here: A fast training step still wastes time if checkpoint writes pause every worker. Singh describes LinkedIn first saving in memory and streaming to storage asynchronously, alongside kernel fusion work. This joins recovery overhead to the earlier compute analysis while keeping proposed distributed restore caches separate from the changes already described as implemented. ### 8. Accelerating Growth Through Optimizing GPU Usage Sahil Khanna, Adobe | 23:53 | AI in Production 2025 Video: https://www.youtube.com/watch?v=SY9c92UfOSU Summary: https://mlopstalks.com/talks/accelerating-growth-through-optimizing-gpu-usage.md Why last: Adobe makes completed jobs the final unit of success. Khanna describes recovering unhealthy hardware and resuming long runs, while placing each job within one cluster and preferably one zone for connectivity. The fleet can span clusters without forcing an individual training run across slow links. End by protecting useful progress as well as accelerating it. ## Editor's note Singh connects faster training with recoverable checkpoints, while Khanna separates job placement from execution in each cluster. ZenML lets teams define Python pipeline steps and choose execution infrastructure through configuration, recording each run's inputs, outputs and code version. This connects a trained artifact to its workflow; the training system still supplies the memory, communication and checkpoint mechanisms discussed here. Written by the MLOps Talks editors (the ZenML team), not by any of the speakers.