Chaos engineering injects bad production conditions in a safe environment so teams can learn how their systems and people respond.
2
Useful experiments begin with an expectation for what the system must continue doing, then test that expectation under failure conditions.
3
Successful experiments can become repeatable regression tests in the CI/CD pipeline after deployments to pre-production systems.
Summary
Benjamin Wilms describes chaos engineering as a way to train systems, teams, and organizations for stressful conditions such as latency spikes, network failures, cloud outages, and deployment problems. The work starts with an expectation, such as customers being able to purchase products even when part of a cloud environment is unavailable. Engineers then create controlled experiments, use observability data to check the effects, and repeat the experiment after making changes. Benjamin distinguishes this approach from the older Chaos Monkey model, which injected failures without checking whether the system was healthy. He also explains how experiments can cover databases, Kafka messages, Kubernetes resources, networks, data distribution, and machine learning data flows. The hardest part is deciding where to start and how to fix what an experiment reveals. Benjamin connects the practice to company culture, blameless learning, customer trust, and the cost of outages. He recommends turning successful experiments into automated CI/CD tests.
Chaos engineering trains systems and people for failure conditions
Benjamin defines chaos engineering around complex, distributed systems whose behavior is hard to understand under stress. Teams inject conditions such as latency spikes, delayed traffic, a network area becoming unavailable, or a failed deployment in a safe environment. The goal is to learn how the technology, the organization, and the people react before a real outage forces the same lesson. He says the customer benefit is direct: customers can continue using the product instead of moving to a competitor when the system has problems.
Smaller startups can use experiments to understand production risk
Demetrios asks whether chaos engineering is useful for startups that do not yet have highly complex systems. Benjamin says a young company may lack the resources to invest heavily in reliability engineering while it focuses on getting its idea into the market. Controlled experiments can still answer a practical question: how much risk does a new feature introduce? A startup can test whether its system can handle a cloud outage or Kubernetes problems before committing the feature to production.
Chaos engineering needs management support and a culture that learns from failure
Benjamin describes a company that used a proof of concept to identify many risks and incidents in its system. Management was impressed by the findings, but did not commit to fixing them. Instead, it asked the team to keep reacting to fires and spend more on people who handled incidents. Benjamin says the problem is organizational as well as technical. Companies need a culture where teams improve after failures, with no finger-pointing or claims that the problem belongs to someone else. Without support and motivation, finding risks does not lead to safer systems.
Experiments can target every layer, including data flows and message ordering
Benjamin says chaos engineering is not limited to hardware or infrastructure. Experiments can inject database read latency, shut down a database, add an instance, and check how quickly data reaches it. Teams can work with Kafka messages by holding some messages back so newer messages pass older ones, which tests how the system handles an update arriving before a create event. Experiments can also target networks, cloud providers, Kubernetes, distributed databases, applications, and resource consumption. The same approach can test machine learning systems whose downstream models depend on live upstream data.
Modern chaos engineering checks system health and stops unsafe experiments
Benjamin contrasts the original Chaos Monkey approach with the current emphasis on directed experiments. Chaos Monkey helped motivate engineering teams because they knew failures could happen at an unknown time. Modern tools should first understand whether the system is healthy enough for an experiment. If the system becomes unhealthy, the experiment should stop immediately and roll back. Benjamin also describes sharing successful experiment collections between teams, so one team's learning about storage failures can help other teams test their own applications.
Every experiment should begin with an expected system behavior
Benjamin says turning off a machine is not the main challenge. Engineers should first define what the system must do and under which conditions it must do it. His example is an e-commerce system that should still let customers purchase products when a cloud provider loses 60 percent of its zones. The experiment then checks whether that expectation is met. A platform can make the work easier, but teams can begin without one by stating the expectation and testing it directly.
Observability must distinguish injected effects from cascading failures
Chaos engineering and observability need to work together. Observability shows how the system reacts to an injected condition, while the chaos tool needs that data to decide whether it is safe to continue. Benjamin gives the example of a CPU or memory attack on one group of machines. If failures appear outside that group, the team may be seeing a cascading failure. The chaos tool should send its own activity to the observability system, and it should consume observability data so it can stop when the wider system begins to fail.
Successful experiments become repeatable CI/CD regression tests
Benjamin says teams should first decide where to start, create experiments, and run them successfully. A successful experiment can then be automated after each deployment in a QA, performance, or pre-production environment. Some experiments encode incidents from the past as regression tests, so the same failure does not return after a later change. Others check nonfunctional requirements under difficult conditions. Repeatability matters because engineers need to compare runs under the same starting conditions and know whether a fix actually changed the result.
Reliability work must be justified with outage cost and customer trust
When Demetrios asks why a company should invest if it only loses a few hours each year, Benjamin answers that hope is not a strategy. He recommends calculating the revenue affected by a ten-minute or one-hour outage. He also points to customer trust. Repeated failures can make customers switch to another app or service, and that trust is not quickly recovered. The argument for chaos engineering therefore depends on the company's own outage cost, customer expectations, and the system behavior it needs to protect.
"The challenge is define your expectations, start from your point of view what is your expectation, what your system needs to do, and then under which conditions."Benjamin Wilms19:31
Who should watch
You run services or data flows where a cloud, network, database, or message-ordering failure could wake someone up during an on-call shift.
Your team has incident history but has not turned those incidents into repeatable tests before production deployments.
You need a practical way to connect reliability experiments with observability, platform engineering, or machine learning data pipelines.
