High-stakes machine learning needs software engineering practices, reliability checks, and accountability because automated decisions can affect people and businesses.
2
The Swiss cheese model explains failures as the alignment of weaknesses across technical systems, procedures, engineering culture, and people.
3
MLOps work should address latent conditions and active failures through monitoring, code review, data and experiment management, versioning, and clear ownership.
Summary
Flavio Clesio argues that the era of one data scientist training a model in a single machine is over for serious machine learning systems. Regulation, especially the right to an explanation under GDPR in Europe, adds pressure to make automated decisions traceable and understandable. He uses James Reason's Swiss cheese model to explain how incidents emerge when weaknesses in several defensive layers line up. Those layers can include architecture, the technology stack, procedures, engineering culture, and people. A missing code review, outdated library, glue code, weak monitoring, or an unsuitable technical choice can remain dormant until an active failure triggers an outage. Flavio describes a recommendation system that returned the same catalog for 17 days because its container ran out of memory and the team lacked suitable observability. He recommends treating MLOps as reliability work, with orchestration, monitoring, experiment management, data versioning, and thoughtful human decisions.
High-stakes ML needs software engineering discipline
Flavio says machine learning platforms are at a turning point. Tools make machine learning easier than they were several years earlier, but many teams still put systems into production without unified testing, code review, or mature code and data management. Regulation in Europe is also pushing companies toward accountability for automated decisions. He describes the old model as one person putting a script in a single machine, and says that approach is over for complex applications. He sees MLOps as the way to bring machine learning work closer to professional software engineering.
Failure stories are missing from much of the MLOps conversation
Flavio criticizes the survivorship bias in machine learning writing. Public accounts often describe a company earning more money after adopting machine learning or using a framework that supposedly solves every problem. They rarely describe projects that failed, teams that were dismissed, systems replaced with rules, or the difficult work of getting code into production. He argues that teams should discuss deployment, code review, data management, and experiment tracking in the projects that struggled. His own text-classification example shows why polished advice can mislead: standard stop-word removal and lemmatization reduced the classifier's performance because words treated as generic stop words identified specific services in the company's domain.
Aviation provides a stricter reliability reference point
Flavio compares machine learning reliability with aviation rather than with the reliability practices of large technology companies. He says aviation improves by learning from past accidents and operates critical aircraft systems at a far higher reliability level. For high-stakes machine learning, reliability includes more than service availability. Teams also need to track the data, code, and model behavior separately. His definition of MLOps is therefore closely tied to making platforms reliable and accountable when their decisions involve other people.
The Swiss cheese model treats incidents as aligned weaknesses
Flavio introduces James Reason's Swiss cheese model, developed in the early 1990s. A complex accident usually has a chain of contributing events rather than one isolated cause. Each defensive layer contains holes, or vulnerabilities. In a machine learning system, layers can include the technology stack, people, processes, and architecture. Latent conditions can sit unnoticed inside those layers, while an active failure is an action or event that exposes the weakness. An incident occurs when holes across several layers line up and create a path through the system.
Latent conditions can wait until a small trigger causes damage
Flavio separates dormant weaknesses from active failures. A latent condition is a condition inside the system, process, or team that does not cause a failure by itself. He uses a glass of water as an example: it is harmless while sitting still, but it can break a laptop if it falls over. In machine learning, missing code review, an old library, weak observability, or a lack of decision-making ownership can remain unnoticed. An active failure can then be a bad code change, an unmonitored pipeline break, or a human action that lets those weaknesses combine.
A recommendation outage involved several failures at once
Flavio describes a recommendation platform that returned the same services or catalog items to customers for 17 days. The immediate symptom was an out-of-memory problem in the container running the application. The deeper account included several contributing factors: the container choice was a poor architectural decision, the person implementing it did not know the platform beforehand, a market campaign increased the item volume, the team lacked a useful heartbeat for the application, and monitoring did not measure recommendation diversity. Increasing the memory limit would have been a local fix. Flavio argues that a postmortem should also examine why the architecture, process, and observability allowed the problem to persist.
Human choices shape reliability as much as tools do
Flavio rejects the idea that MLOps can remove the human factor. Culture depends on people, and tools only help when someone knows how to use them and can improve the process around them. He warns about resume-driven development, where a developer chooses a technology to add it to a resume instead of fitting the team's needs. One example used Scala for a team that worked in Python, making later maintenance take much longer. He also describes a pipeline that used Elixir to read spreadsheets without sharing the risks with the company. A team needs decision makers, shared standards, and ownership alongside automation.
MLOps tooling covers several distinct operational needs
Flavio groups MLOps tools by the work they support. Orchestration coordinates data flows, training, and pipeline dependencies, with Airflow and Kubeflow among his examples. Observability covers logs, traces, system metrics, and application measures such as recommendation diversity. Experiment management records experiments so teams can reproduce them, retrain models, and roll back when needed. Data versioning and management address the problem of freezing datasets in storage without a reliable way to identify or reproduce them. He mentions DVC and Pachyderm for data versioning, and he says Airflow improved transparency over Jenkins for one of his pipelines, although he considers Airflow over-engineered for simple jobs.
"Instead of keeping swatting against mosquitoes, if you have some kind of mosquito problem in your home, just go to the swamp and destroy the swamp where they breed."Flavio Clesio39:40
Who should watch
You operate recommendation, scoring, classification, or other machine learning systems whose failures affect customers or business decisions.
Your team has models in production but lacks consistent code review, monitoring, experiment tracking, or data versioning.
You are choosing an orchestration or observability tool and want to consider team skills, maintenance, and failure analysis alongside features.
