Large-scale LLM training can fail through hardware faults, unstable software, data-loading problems, and silent accuracy issues.
2
Teams should define use-case-specific evaluations before training, then track those evaluations alongside general benchmarks and scaling laws.
3
The simplest system that meets the need is usually the right starting point, with complexity added only when privacy, cost, latency, or quality requires it.
Summary
Davis Blalock and Bandish Shah describe LLM training as a system with failure points at every layer. GPUs can fail silently, distributed communication can time out, object stores can be overloaded by checkpoint traffic, and software versions can break assumptions in ways that are hard to diagnose. Accuracy failures are harder still because the run may produce plausible numbers without explaining what went wrong. They argue that evaluation must come before training, with metrics tied to the actual business problem and regular checks during the run. For model development, Davis recommends starting with the simplest workable option, then adding a private model, prompting, retrieval, fine-tuning, or pre-training only when needed. The conversation also covers tokenization bugs, data shuffling, resumption, deduplication, and manual data inspection. Bandish explains why tested configurations and good defaults are valuable to customers who cannot afford failed training runs.
Training at scale fails through hardware faults that may produce no useful error
Davis Blalock says GPU failures are common in large training jobs, and the result may be a crash, a slow node, or subtly incorrect outputs. A NCCL timeout only indicates that something failed to respond during distributed communication. Engineers may need to run diagnostics across every node, while some problems are transient. Checkpointing can fail when many nodes overload an object store, and network file systems can buckle under the combined load. These failures are hard because the system often gives an indirect symptom instead of a precise hardware exception.
Deep learning accuracy is harder to debug than ordinary system failures
Davis Blalock calls debugging deep learning accuracy the worst debugging problem in computer science because low numbers do not identify a cause. There is no exception that points to the broken component. Bandish Shah adds that a model can retain good accuracy even when the system is computing the wrong thing, then behave differently after an underlying change. Their work therefore requires disciplined, systematic investigation, along with experience and occasional guesswork. A training run can appear operational while still producing a model that misses its intended target.
Evaluation of the real use case has to come before an expensive training run
Bandish Shah says the first customer question should be how they will know that a model solves their problem. Leaderboard metrics may not reflect code generation, call-center automation, or another specific business task. Teams should define their own evaluations and use them alongside general benchmarks. Davis Blalock describes a chicken-and-egg problem because some capabilities cannot be judged until a model exists, so teams also track progress with scaling laws and frequent evaluations. They use informal checks as well, such as asking whether a model can identify who trained it.
A complete working configuration can protect customers from costly failed runs
Davis Blalock says customers value a configuration that includes images, hyperparameters, and the other pieces needed to train successfully. It lets them point a known setup at their data instead of discovering every compatibility and tuning issue themselves. He compares this to insurance for a training run, especially for a startup that has raised $10 million and may spend $6 million training a model. Bandish Shah explains that MosaicML turns its own hard-won training experience into a product so customers do not have to repeat the same debugging work.
Model development should add complexity only when the simpler option fails
Davis Blalock recommends starting with a third-party API when it meets the use case. Privacy, cost, or latency constraints may justify deploying an existing model. Teams can then try better prompts with in-context examples, retrieval, fine-tuning, a larger model, or their own pre-training as needed. Bandish Shah says customers often begin with smaller models and work upward through small gating runs. This helps them predict what a larger run may achieve and manage the cost of experimentation. The choice depends on privacy, intellectual property, quality, cost, performance, and deployment needs.
Tokenization and distributed data loading create their own model failures
Davis Blalock explains that LLMs train on tokens rather than raw text, and tokenizers can create invalid bytes or unexpected combinations of whitespace and punctuation. A Falcon math evaluation produced garbage until the team discovered that a trailing space was required. Data loading also becomes difficult at scale. Nodes must avoid redundant requests, remote object storage can be slow, and poor shuffling can create oscillating loss curves, which the team calls 'wavy boys.' Resuming after a crash raises further questions about redownloading data and preserving shuffle behavior across a different number of nodes.
Data quality still requires people to inspect samples directly
Davis Blalock says there is no substitute for looking at the data. Preprocessing may remove every table from a document while leaving references such as 'table one,' and datasets contain problems that differ by collection and task. Some filtering can be tested by training a small model, but useful signal may require a large model and substantial compute. Bandish Shah describes pipelines for deduplication and filtering that once took days and became much faster after help from Databricks data engineers and Spark specialists. Quality work includes selecting the right data for pre-training, instruction tuning, prompts, responses, and synthetic examples.
"If you want to remember what it's like to train LLMs, you just have to remember there's a song written about it: it's 10% luck, 20% skill, 15% concentrated power of will, 5% pleasure and 50% pain."Davis Blalock26:33
Who should watch
You are planning a large pre-training run and need to understand where hardware, software, checkpointing, and data pipelines can fail.
Your team is unsure whether to use an API, retrieval, fine-tuning, or a newly trained model for a specific product problem.
You need evaluation methods for a domain-specific model and want a realistic account of why data quality and accuracy debugging are expensive.
