# High Stakes ML: Active Failures, Latent Factors

Flavio Clesio, MyHammer AG | MLOps Meetup | Episode 5 | 55:27
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=9g4deV1uNZo
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/high-stakes-ml-active-failures-latent-factors
Published: 2020-04-16
Tags: monitoring, observability, orchestration, reliability

## TL;DR
- High-stakes machine learning needs software engineering practices, reliability checks, and accountability because automated decisions can affect people and businesses.
- The Swiss cheese model explains failures as the alignment of weaknesses across technical systems, procedures, engineering culture, and people.
- MLOps work should address latent conditions and active failures through monitoring, code review, data and experiment management, versioning, and clear ownership.

## Summary
Flavio Clesio argues that the era of one data scientist training a model in a single machine is over for serious machine learning systems. Regulation, especially the right to an explanation under GDPR in Europe, adds pressure to make automated decisions traceable and understandable. He uses James Reason's Swiss cheese model to explain how incidents emerge when weaknesses in several defensive layers line up. Those layers can include architecture, the technology stack, procedures, engineering culture, and people. A missing code review, outdated library, glue code, weak monitoring, or an unsuitable technical choice can remain dormant until an active failure triggers an outage. Flavio describes a recommendation system that returned the same catalog for 17 days because its container ran out of memory and the team lacked suitable observability. He recommends treating MLOps as reliability work, with orchestration, monitoring, experiment management, data versioning, and thoughtful human decisions.

## Key ideas
### High-stakes ML needs software engineering discipline
[05:30](https://www.youtube.com/watch?v=9g4deV1uNZo&t=330s)
Flavio says machine learning platforms are at a turning point. Tools make machine learning easier than they were several years earlier, but many teams still put systems into production without unified testing, code review, or mature code and data management. Regulation in Europe is also pushing companies toward accountability for automated decisions. He describes the old model as one person putting a script in a single machine, and says that approach is over for complex applications. He sees MLOps as the way to bring machine learning work closer to professional software engineering.

### Failure stories are missing from much of the MLOps conversation
[09:06](https://www.youtube.com/watch?v=9g4deV1uNZo&t=546s)
Flavio criticizes the survivorship bias in machine learning writing. Public accounts often describe a company earning more money after adopting machine learning or using a framework that supposedly solves every problem. They rarely describe projects that failed, teams that were dismissed, systems replaced with rules, or the difficult work of getting code into production. He argues that teams should discuss deployment, code review, data management, and experiment tracking in the projects that struggled. His own text-classification example shows why polished advice can mislead: standard stop-word removal and lemmatization reduced the classifier's performance because words treated as generic stop words identified specific services in the company's domain.

### Aviation provides a stricter reliability reference point
[13:46](https://www.youtube.com/watch?v=9g4deV1uNZo&t=826s)
Flavio compares machine learning reliability with aviation rather than with the reliability practices of large technology companies. He says aviation improves by learning from past accidents and operates critical aircraft systems at a far higher reliability level. For high-stakes machine learning, reliability includes more than service availability. Teams also need to track the data, code, and model behavior separately. His definition of MLOps is therefore closely tied to making platforms reliable and accountable when their decisions involve other people.

### The Swiss cheese model treats incidents as aligned weaknesses
[16:19](https://www.youtube.com/watch?v=9g4deV1uNZo&t=979s)
Flavio introduces James Reason's Swiss cheese model, developed in the early 1990s. A complex accident usually has a chain of contributing events rather than one isolated cause. Each defensive layer contains holes, or vulnerabilities. In a machine learning system, layers can include the technology stack, people, processes, and architecture. Latent conditions can sit unnoticed inside those layers, while an active failure is an action or event that exposes the weakness. An incident occurs when holes across several layers line up and create a path through the system.

### Latent conditions can wait until a small trigger causes damage
[19:11](https://www.youtube.com/watch?v=9g4deV1uNZo&t=1151s)
Flavio separates dormant weaknesses from active failures. A latent condition is a condition inside the system, process, or team that does not cause a failure by itself. He uses a glass of water as an example: it is harmless while sitting still, but it can break a laptop if it falls over. In machine learning, missing code review, an old library, weak observability, or a lack of decision-making ownership can remain unnoticed. An active failure can then be a bad code change, an unmonitored pipeline break, or a human action that lets those weaknesses combine.

### A recommendation outage involved several failures at once
[21:47](https://www.youtube.com/watch?v=9g4deV1uNZo&t=1307s)
Flavio describes a recommendation platform that returned the same services or catalog items to customers for 17 days. The immediate symptom was an out-of-memory problem in the container running the application. The deeper account included several contributing factors: the container choice was a poor architectural decision, the person implementing it did not know the platform beforehand, a market campaign increased the item volume, the team lacked a useful heartbeat for the application, and monitoring did not measure recommendation diversity. Increasing the memory limit would have been a local fix. Flavio argues that a postmortem should also examine why the architecture, process, and observability allowed the problem to persist.

### Human choices shape reliability as much as tools do
[30:40](https://www.youtube.com/watch?v=9g4deV1uNZo&t=1840s)
Flavio rejects the idea that MLOps can remove the human factor. Culture depends on people, and tools only help when someone knows how to use them and can improve the process around them. He warns about resume-driven development, where a developer chooses a technology to add it to a resume instead of fitting the team's needs. One example used Scala for a team that worked in Python, making later maintenance take much longer. He also describes a pipeline that used Elixir to read spreadsheets without sharing the risks with the company. A team needs decision makers, shared standards, and ownership alongside automation.

### MLOps tooling covers several distinct operational needs
[46:34](https://www.youtube.com/watch?v=9g4deV1uNZo&t=2794s)
Flavio groups MLOps tools by the work they support. Orchestration coordinates data flows, training, and pipeline dependencies, with Airflow and Kubeflow among his examples. Observability covers logs, traces, system metrics, and application measures such as recommendation diversity. Experiment management records experiments so teams can reproduce them, retrain models, and roll back when needed. Data versioning and management address the problem of freezing datasets in storage without a reliable way to identify or reproduce them. He mentions DVC and Pachyderm for data versioning, and he says Airflow improved transparency over Jenkins for one of his pipelines, although he considers Airflow over-engineered for simple jobs.

## Notable quotes
- Flavio Clesio: "The era of a single guy with the same script inside a machine, it's over completely right now." (07:30)
- Flavio Clesio: "If I need to define MLOps in a single word, for me at least it's reliability." (15:50)
- Flavio Clesio: "It's not only because of a single factor, it's because when aligning a perfect storm between several different layers." (26:27)
- Flavio Clesio: "Everything is about the human factors in the end of the day." (32:13)
- Flavio Clesio: "Instead of keeping swatting against mosquitoes, if you have some kind of mosquito problem in your home, just go to the swamp and destroy the swamp where they breed." (39:40)

## Tools & references mentioned
- MyHammer AG
- James Reason
- MLOps Community
- GDPR
- Airflow
- Kubeflow
- Jenkins
- DVC
- Pachyderm
- ElasticSearch
- Kibana
- Prometheus
- Sentry
- Datadog
- MLflow
- Polyaxon
- Google Cloud Platform
- SageMaker
- Data Science
- Algorithmia
- H2O.ai
- Paperspace
- BigQuery
- PyTorch
- VGG-16
- Scala
- Elixir

## Who should watch
- You operate recommendation, scoring, classification, or other machine learning systems whose failures affect customers or business decisions.
- Your team has models in production but lacks consistent code review, monitoring, experiment tracking, or data versioning.
- You are choosing an orchestration or observability tool and want to consider team skills, maintenance, and failure analysis alongside features.

## Editor's note

Flavio Clesio says that the era of one data scientist training a model on a single machine is over for serious machine learning systems. ZenML records each run's steps, inputs, outputs, and code version, so a model can be traced back to the data and code that produced it. This gives teams a concrete record when they need to reproduce or investigate a pipeline.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Reliable Machine Learning](https://mlopstalks.com/talks/reliable-machine-learning) (Niall Murphy, Stanza Systems & Todd Underwood, Google, 1:02:25)
- [MLOps in Practice: Common Challenges and Lessons Learned](https://mlopstalks.com/talks/mlops-in-practice-common-challenges-and-lessons-learned) (Marouen Hizaoui & Mo Basirati, Machine Learning Reply, 1:03:01)
- [Enterprises Using MLOps, the Changing LLM Landscape, MLOps Pipelines](https://mlopstalks.com/talks/enterprises-using-mlops-the-changing-llm-landscape-mlops-pipelines) (Chris Van Pelt, Weights & Biases, 47:51)
- [The Godfather Of MLOps](https://mlopstalks.com/talks/the-godfather-of-mlops) (D. Sculley, Google, 51:25)
- [Practical MLOps Part 2](https://mlopstalks.com/talks/practical-mlops-part-2) (Alfredo Deza, Author and Speaker, 1:01:38)