LLM testing should cover the base model, the application as users experience it, and real-world data before and after production.
2
Red teaming should combine open-ended manual exploration with focused automated tests for specific failures and data transformations.
3
Prompt injection, prompt extraction, harmful outputs, hallucinations, inconsistent answers, and data privacy are examples of risks that stress tests can probe.
Summary
Finn Howell presents red teaming as a way to find deficiencies in AI systems and produce recommendations for addressing them. She argues that teams should test isolated base models and complete applications through their normal user experience, using real-world data before deployment and continuing after release. Her framework groups failures into operational, ethical, and security risks. Examples include hallucinations, irrelevant answers, harmful or biased outputs, prompt injection, prompt extraction, and data privacy problems. Howell describes a cycle in which manual exploration produces tests that teams then automate. Open-ended testing looks for failures across system components, while focused tests probe particular inputs and expected behaviors. She shows prompt injection and extraction tests, data transformations such as lowercasing and character changes, and alignment tests using misinformation, sexist, and racist prompts. She is also direct about the difficulty of deciding whether an output has passed and the cost of running large numbers of model evaluations.
AI red teaming tests whole systems for actionable deficiencies
Howell describes traditional red teaming as simulating adversarial conditions to improve defenses. For AI, the input is an AI system and the output is a set of deficiencies with recommendations for action. The scope now includes more than cybersecurity. Teams must test how an AI system behaves under attacks and unexpected inputs. She says this work is becoming more important as the White House, NIST, and the EU push for external security testing and general validation of models.
Testing must cover the model, the user-facing application, and production
Howell says teams should evaluate an isolated base model, then test the application through its normal user experience with real-world data. Testing should continue after deployment, especially when a company or use case is exposed to security risks and adversarial attacks. Attackers change their methods over time, so a pre-production assessment cannot be the only check. Her lifecycle view covers both the model and the surrounding application.
Open-ended exploration and focused tests find different failures
Howell separates broad, open-ended red teaming from focused, depth-first testing. Open-ended work explores a wide space of possible failures and looks for problems across system components. Focused work targets a specific failure or data point, much like a software unit test. She groups common risks into operational, ethical, and security categories. Examples include hallucinations, inconsistent or irrelevant answers, harmful outputs, biased or exclusive behavior, prompt injection, data privacy, and supply-chain issues.
Howell recommends automating what can be automated while retaining manual exploration. She compares system-level manual testing with unit tests that repeatedly probe a model. Automated red teaming can be framed as an optimization problem: given an existing data point, find a nearby data point that causes a different model response. LLMs make this difficult because they produce many possible responses, and the search can become hard to manage. Running many model calls also creates a cost problem that teams must account for.
Prompt injection tests can target different parts of a prompt
Howell shows a prompt injection test that adds an attack to user input, context, or the question. In one example, the model handles an injection placed in the context, but falls for an injection placed in the question and returns the phrase indicating that the jailbreak succeeded. The example is deliberately simple, yet the same weakness could cause a model to produce toxic language or false information. Turning the attack into a unit test requires a way to judge whether the output passes or fails.
Prompt extraction tests whether system instructions can be recovered
In a prompt extraction test, the evaluator tries to recover the model's original instructions, including its system prompt. Howell describes placing a random token in the system prompt and checking whether the model reveals it. In her example, the token can be extracted. This gives the test a concrete success condition, although the broader challenge remains deciding how to evaluate model outputs reliably across different attacks.
Small data changes can expose unstable model behavior
Howell uses data transformation tests to check whether minor input changes produce unexpected output changes. Lowercasing an input causes a substantially different response in her example. Other transformations include swapping characters, simulating OCR errors, adding characters, and adding synonyms. The evaluator compares the responses before and after the transformation with similarity measures. Howell says it remains difficult to decide how similar the text should be and when a difference means the test has failed.
Howell describes feeding models prompts involving misinformation, sexist content, and racist content, sometimes with many examples of the behavior. The test checks whether the model follows the pattern or refuses to answer. In her example, the model answers a sexist question in line with the supplied examples, producing toxic output instead of rejecting the request. She uses this alongside security tests to show how automated evaluations can examine ethical behavior as well as technical defenses.
"We think that you should test both your isolated base model but also ensure you're testing your application through the normal user experience and with real world data."02:00
Who should watch
You are responsible for validating an LLM application and need a testing plan that continues after deployment.
Your model can receive untrusted input, and you need concrete tests for prompt injection, prompt extraction, or harmful outputs.
You are building automated evaluations and need to think about output scoring, input transformations, search difficulty, and model-call cost.
