# Build Reliable Systems with Chaos Engineering

Benjamin Wilms, Steadybit | MLOps Podcast | Episode 237 | 46:58
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=hUXVTLfkSUU
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/build-reliable-systems-with-chaos-engineering
Published: 2024-05-31
Tags: observability, reliability, testing

## TL;DR
- Chaos engineering injects bad production conditions in a safe environment so teams can learn how their systems and people respond.
- Useful experiments begin with an expectation for what the system must continue doing, then test that expectation under failure conditions.
- Successful experiments can become repeatable regression tests in the CI/CD pipeline after deployments to pre-production systems.

## Summary
Benjamin Wilms describes chaos engineering as a way to train systems, teams, and organizations for stressful conditions such as latency spikes, network failures, cloud outages, and deployment problems. The work starts with an expectation, such as customers being able to purchase products even when part of a cloud environment is unavailable. Engineers then create controlled experiments, use observability data to check the effects, and repeat the experiment after making changes. Benjamin distinguishes this approach from the older Chaos Monkey model, which injected failures without checking whether the system was healthy. He also explains how experiments can cover databases, Kafka messages, Kubernetes resources, networks, data distribution, and machine learning data flows. The hardest part is deciding where to start and how to fix what an experiment reveals. Benjamin connects the practice to company culture, blameless learning, customer trust, and the cost of outages. He recommends turning successful experiments into automated CI/CD tests.

## Key ideas
### Chaos engineering trains systems and people for failure conditions
[03:09](https://www.youtube.com/watch?v=hUXVTLfkSUU&t=189s)
Benjamin defines chaos engineering around complex, distributed systems whose behavior is hard to understand under stress. Teams inject conditions such as latency spikes, delayed traffic, a network area becoming unavailable, or a failed deployment in a safe environment. The goal is to learn how the technology, the organization, and the people react before a real outage forces the same lesson. He says the customer benefit is direct: customers can continue using the product instead of moving to a competitor when the system has problems.

### Smaller startups can use experiments to understand production risk
[06:23](https://www.youtube.com/watch?v=hUXVTLfkSUU&t=383s)
Demetrios asks whether chaos engineering is useful for startups that do not yet have highly complex systems. Benjamin says a young company may lack the resources to invest heavily in reliability engineering while it focuses on getting its idea into the market. Controlled experiments can still answer a practical question: how much risk does a new feature introduce? A startup can test whether its system can handle a cloud outage or Kubernetes problems before committing the feature to production.

### Chaos engineering needs management support and a culture that learns from failure
[07:54](https://www.youtube.com/watch?v=hUXVTLfkSUU&t=474s)
Benjamin describes a company that used a proof of concept to identify many risks and incidents in its system. Management was impressed by the findings, but did not commit to fixing them. Instead, it asked the team to keep reacting to fires and spend more on people who handled incidents. Benjamin says the problem is organizational as well as technical. Companies need a culture where teams improve after failures, with no finger-pointing or claims that the problem belongs to someone else. Without support and motivation, finding risks does not lead to safer systems.

### Experiments can target every layer, including data flows and message ordering
[12:03](https://www.youtube.com/watch?v=hUXVTLfkSUU&t=723s)
Benjamin says chaos engineering is not limited to hardware or infrastructure. Experiments can inject database read latency, shut down a database, add an instance, and check how quickly data reaches it. Teams can work with Kafka messages by holding some messages back so newer messages pass older ones, which tests how the system handles an update arriving before a create event. Experiments can also target networks, cloud providers, Kubernetes, distributed databases, applications, and resource consumption. The same approach can test machine learning systems whose downstream models depend on live upstream data.

### Modern chaos engineering checks system health and stops unsafe experiments
[14:37](https://www.youtube.com/watch?v=hUXVTLfkSUU&t=877s)
Benjamin contrasts the original Chaos Monkey approach with the current emphasis on directed experiments. Chaos Monkey helped motivate engineering teams because they knew failures could happen at an unknown time. Modern tools should first understand whether the system is healthy enough for an experiment. If the system becomes unhealthy, the experiment should stop immediately and roll back. Benjamin also describes sharing successful experiment collections between teams, so one team's learning about storage failures can help other teams test their own applications.

### Every experiment should begin with an expected system behavior
[19:31](https://www.youtube.com/watch?v=hUXVTLfkSUU&t=1171s)
Benjamin says turning off a machine is not the main challenge. Engineers should first define what the system must do and under which conditions it must do it. His example is an e-commerce system that should still let customers purchase products when a cloud provider loses 60 percent of its zones. The experiment then checks whether that expectation is met. A platform can make the work easier, but teams can begin without one by stating the expectation and testing it directly.

### Observability must distinguish injected effects from cascading failures
[24:47](https://www.youtube.com/watch?v=hUXVTLfkSUU&t=1487s)
Chaos engineering and observability need to work together. Observability shows how the system reacts to an injected condition, while the chaos tool needs that data to decide whether it is safe to continue. Benjamin gives the example of a CPU or memory attack on one group of machines. If failures appear outside that group, the team may be seeing a cascading failure. The chaos tool should send its own activity to the observability system, and it should consume observability data so it can stop when the wider system begins to fail.

### Successful experiments become repeatable CI/CD regression tests
[32:23](https://www.youtube.com/watch?v=hUXVTLfkSUU&t=1943s)
Benjamin says teams should first decide where to start, create experiments, and run them successfully. A successful experiment can then be automated after each deployment in a QA, performance, or pre-production environment. Some experiments encode incidents from the past as regression tests, so the same failure does not return after a later change. Others check nonfunctional requirements under difficult conditions. Repeatability matters because engineers need to compare runs under the same starting conditions and know whether a fix actually changed the result.

### Reliability work must be justified with outage cost and customer trust
[43:08](https://www.youtube.com/watch?v=hUXVTLfkSUU&t=2588s)
When Demetrios asks why a company should invest if it only loses a few hours each year, Benjamin answers that hope is not a strategy. He recommends calculating the revenue affected by a ten-minute or one-hour outage. He also points to customer trust. Repeated failures can make customers switch to another app or service, and that trust is not quickly recovered. The argument for chaos engineering therefore depends on the company's own outage cost, customer expectations, and the system behavior it needs to protect.

## Notable quotes
- Benjamin Wilms: "With chaos engineering you can inject those bad moments, but in a safe environment." (04:12)
- Benjamin Wilms: "The challenge is define your expectations, start from your point of view what is your expectation, what your system needs to do, and then under which conditions." (19:31)
- Benjamin Wilms: "No tool on that market or maybe in the future will be able to tell you exactly, in a complex system, how you can fix it." (27:39)
- Benjamin Wilms: "Hope is not a strategy." (43:43)

## Tools & references mentioned
- Steadybit
- Chaos Monkey
- Chaos Monkey for Spring Boot
- Spring Boot
- AWS
- AWS Trainium
- AWS Inferentia
- Kubernetes
- Kafka
- Redis

## Who should watch
- You run services or data flows where a cloud, network, database, or message-ordering failure could wake someone up during an on-call shift.
- Your team has incident history but has not turned those incidents into repeatable tests before production deployments.
- You need a practical way to connect reliability experiments with observability, platform engineering, or machine learning data pipelines.

## Editor's note

From the pack [Testing machine learning systems](https://mlopstalks.com/packs/testing-machine-learning-systems):

Elgendy's scenario history and Sragner's insistence on testing the deployed implementation both depend on knowing which code and data produced a result. ZenML records pipeline steps, inputs, outputs and code versions for each run. That gives a team's model tests a traceable place in the workflow, while the team still defines the cases and pass criteria.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [SRE for ML Infra](https://mlopstalks.com/talks/sre-for-ml-infra) (Todd Underwood, Google, 1:11:50)
- [Machine Learning SRE](https://mlopstalks.com/talks/machine-learning-sre) (Niall Murphy, Microsoft Azure, 48:29)
- [High Stakes ML: Active Failures, Latent Factors](https://mlopstalks.com/talks/high-stakes-ml-active-failures-latent-factors) (Flavio Clesio, MyHammer AG, 55:27)
- [Reliable Machine Learning](https://mlopstalks.com/talks/reliable-machine-learning) (Niall Murphy, Stanza Systems & Todd Underwood, Google, 1:02:25)
- [Scaling Data Reliably: A Journey in Growing Through Data Pain Points](https://mlopstalks.com/talks/scaling-data-reliably-a-journey-in-growing-through-data-pain-points) (Miriah Peterson, 16:04)