ML infrastructure should make useful models available to ordinary software engineers, rather than focusing only on new algorithms.
2
Teams can start with a narrow, reliable serving system and define clear interfaces around each part of the pipeline.
3
Most serious ML outages are caused by distributed-systems and operational failures, so reliability teams need strong systems skills and a blameless culture.
Summary
Todd Underwood argues that machine learning infrastructure is still at an early stage. The industry has developed advanced algorithms faster than it has packaged them into services that ordinary engineers can use. His goal is to let a software engineer with an idea and some data build a functional experiment in a production-ready environment within a week or two. He recommends starting with narrow use cases, building reliable components, and defining clear interfaces between them. Teams do not need to solve the entire pipeline at once. Todd also explains that most serious failures in an ML system are ordinary distributed-systems problems, such as scheduling, memory, throughput, and deployment errors. ML adds data-dependent failure modes, where a small change can harm one population while leaving aggregate metrics unchanged. He connects reliability work to hiring, blameless retrospectives, fairness, ethics, and the industry's need to earn trust from organizations that fear opaque systems.
ML infrastructure should make useful technology accessible to ordinary engineers
Todd says the industry pays too much attention to new algorithms while useful techniques remain difficult to use outside large technology companies. A model that is not packaged into software or a service is hard for normal teams to apply. He wants a junior software engineer, after learning the basics of a new role, to take an idea and some data and produce a functional experimental model in a production-ready environment within a week or two. He describes productionization as a separate task that can enable new applications, rather than merely making existing systems reliable.
A complete ML pipeline can be built by breaking it into defined components
Todd breaks the path from an idea to a served model into a set of concrete systems. A team needs data acquisition, metadata, feature storage, model storage, model metadata, training, quality analysis, gating, and serving. Each part can be difficult, but each is still a defined engineering problem. He says progress depends on choosing a narrower group of common use cases instead of insisting that the first system handle everything. Product decisions are needed to decide which needs to support first.
A reliable serving system can be the right place to start
Todd strongly agrees with starting to serve a model before every part of the pipeline is polished. At Google, one internal effort began with the serving system because teams already had TensorFlow training frameworks and bespoke processes that produced saved models. The new service copied models around the world and answered lookups with high throughput and reliability. It assumed that upstream systems existed, which kept the initial problem narrow. Todd says the main value was the API and the interfaces, because teams could replace an individual box while still meeting the contract before and after it.
ML reliability teams need systems skills more than formal ML credentials
Todd says most people working on ML reliability at Google do not have a machine learning background. He estimates that only around five to eight percent have graduate-level ML education. Reliability engineers can learn the specific ways ML pipelines fail, such as sensitivity to changes in data distributions. They also need to understand data storage, distributed processing, and serving systems. Todd hires people who are able and eager to learn rather than selecting only for knowledge of a particular framework or academic specialization.
ML failures can affect a small population while aggregate metrics look fine
Todd explains that ML systems depend more heavily on the data itself than many conventional processing systems. Dropping all data from Latin America might remove only six percent of the total data, yet produce wrong conclusions about vocabulary, purchases, and searches for Spanish-speaking users. English-speaking users in Canada might see no problem at all. This differs from changing a value in a tax calculation, where the error is easier to detect. Teams therefore need to inspect slices of data and understand how changes in the pipeline affect particular groups.
Most serious ML outages are ordinary distributed-systems failures
Todd and Daniel Papasian reviewed more than ten years of outage postmortems for one system, while acknowledging that the sample may not represent every ML service. The majority of outages, including the majority of serious ones, had nothing specifically to do with machine learning. They involved issues such as memory errors, CPU bugs at very large scale, scheduling problems, processes that failed to start, and throughput or orchestration failures. Todd's conclusion is that ML reliability teams need people who can reason about distributed systems, not only people who can build models.
Blameless postmortems turn failures into system improvements
Todd recommends replacing blame with a systems approach. When a feature definition changes in training but not in serving, the person may have made a simple mistake, yet the deeper question is why the system made that mistake easy. He compares unsafe tooling to placing a large red button on someone's desk and then blaming them for pressing it. Teams should remove unsafe paths, add safeguards, and admit when limited investment means a related failure may happen again. Cross-team analysis can reveal shared weaknesses in infrastructure that no single product team would fix alone.
ML services must earn trust through clear guarantees and accountability
Todd says many organizations lack both the staff to build integrated ML systems and the trust needed to deploy them. Senior leaders may see these systems as opaque, and failures can damage a company's finances or reputation. He expects providers to operate applications on behalf of customers and accept responsibility when they fail. He also wants model-quality assertions that can guarantee a new model will not ruin a business, even if the guarantee cannot promise perfection or improvement in every respect. He connects this work to explaining risks clearly to executives, legislators, and the public.
"When do you want to blame someone, you want to blame someone who intentionally goes around the systems repeatedly to try to create outages or create unnecessary risk."Todd Underwood42:35
Who should watch
You are building ML infrastructure and need to decide which part of the pipeline to make reliable first.
Your background is in systems or software engineering, and you want to understand how much formal machine learning knowledge an ML reliability role requires.
You lead an engineering team that needs better postmortems, clearer ownership, or a more honest way to discuss ML risks with executives and policymakers.
