SRE principles still apply to machine learning, but service-level objectives must account for model quality, confidence, data, and other ML-specific behavior.
2
Production awareness should begin early in a team, even before an organization hires a dedicated SRE.
3
Machine learning incident response needs clear escalation paths, more time for model degradation, and a fast rollback mechanism.
Summary
Niall Murphy explains how site reliability engineering applies to machine learning systems. The basic responsibilities remain familiar: define acceptable service levels, measure production behavior, react when thresholds are crossed, and feed operational information back into system design. The hard part is that an ML service can fail without returning an error. A classifier may keep responding while its confidence or output quality declines, so SLOs need to include ML-specific measures. Niall argues that production-minded behavior should start with the first developers rather than waiting for a dedicated SRE team. He describes Google's historical use of roughly five to ten percent of engineering effort for SRE, while stressing that the right level depends on the service. The conversation also covers monitoring, team boundaries, incident response, and model rollback. Niall recommends a fixed, context-free "big red button" that returns a system to a known good model when teams cannot diagnose degradation quickly.
SRE principles stay stable while ML changes what must be measured
Niall says the basic SRE principles remain in place for machine learning. Teams still need to care about the service, define thresholds for action, and use production feedback to change the design. The details differ because an ML system can return a technically valid response while its quality declines. In a classifier, for example, confidence might fall from an average of 0.8 to 0.5 without producing a conventional failure. An SLO therefore cannot only count failed calls or HTTP errors. It must describe the behavior that matters to users and the business.
SLOs need an organizational agreement and an ML-specific technical measure
Niall separates SLOs into an organizational and a technical part. The organization must agree on the level of service users should receive. Assuming 100 percent availability creates an impossible target, while treating the current level as the target may accept poor service. The technical part defines the number, the measurement method, and the metrics. In ML, that work is still developing because teams must decide what quality means for a model. Demetrios adds that SLOs can act as a contract with data sources, such as an expectation that feature data will not regularly arrive with missing values.
Production awareness should begin before a dedicated SRE team exists
Niall avoids giving a company-size rule for hiring its first SRE. He says teams should instead ask which behaviors are needed to run a good service. Production-oriented behavior should begin early because systems are cheaper to make observable, scalable, and maintainable before they become difficult to change. Even a single developer benefits from exposure to production. He gives Google's historical ratio as roughly five to ten percent of engineering effort for SRE, while noting that the right amount depends on the service. A system that naturally delivers three nines needs a different response from one that cannot reach one nine.
SRE responsibilities can grow from people who care about production
Niall describes an early Google pattern in which engineers self-selected around production performance. They asked questions such as why an index had not been rebuilt for months and gradually became a subject-matter team. He thinks organizations can often find this behavior among a group of engineers before creating formal SRE roles. David describes a similar path from data science into MLOps, driven by the difficulty of putting models into production. Niall still sees value in a formal team because titles, support, and recognition can help people develop and pass on these practices.
ML monitoring must cover the whole system and the full model life cycle
Niall says SREs and data scientists often use the idea of monitoring differently. Data scientists may focus on model metrics, KPIs, and a carefully prepared evaluation dataset. SREs tend to ask for visibility across the model's whole life cycle, including production. Data scientists bring knowledge about distributions, training, data availability, and how inputs affect model behavior. SREs bring systems thinking across services, interfaces, networks, access patterns, and infrastructure. The conversation argues for combining these views rather than assigning each concern to a separate team and throwing the model over a wall.
ML incidents can involve business conditions and take much longer to resolve
Niall says incident response is more complicated for ML because models connect to more parts of the business. Investigating a problem may require checking whether a commodity price changed, whether a supplier can provide an item, or whether that change caused over-purchasing or under-purchasing. Such investigations can take days or weeks, unlike the roughly two-hour median incident resolution time he associates with cloud providers and cloud consumers. A data engineer or software engineer may be able to restart a job, but a data scientist may be needed to assess the statistical quality of its inputs and outputs.
A big red button gives teams a safe first response to model degradation
Niall describes two approaches to ML on-call. Google made escalation more acceptable because complex model problems were not always practical for an SRE team to master quickly. Teams also allowed more response time because model degradation usually happens more slowly than a web service outage. The second measure was a "big red button," a relatively context-free mechanism that can quickly return production to a known good state. Usually this means model rollback. If a schema change makes rollback impossible, the emergency mechanism must instead build and deploy a corrected version quickly.
Company culture changes how ML operations practices spread
Niall contrasts Google and Microsoft Azure. He describes Google as a company of systems and rules, where work that could be expressed as an extension of existing systems could often proceed without much discussion. He describes Microsoft as a company of humans, where work generally requires conversations with people. As a result, Google could respond with centralized teams and extensions of established SRE behavior, while Microsoft business units made more local choices based on customer needs. Across the industry, he sees ML adoption growing quickly without equally mature processes and standards.
"The thing that matters is creating what's called a big red button, a fixed, relatively context-free mechanism that you can just mash heavily with your hand and have that do something useful in production."Niall Murphy32:26
Who should watch
Your ML service returns predictions successfully, but you do not know how to define or measure acceptable model quality in production.
Your team is deciding whether to create a dedicated SRE or MLOps role and needs a behavior-based way to think about that decision.
You are designing on-call for models and need an escalation path and rollback action that work when the responder cannot diagnose model degradation.
