Agent evaluation starts with a curated set of test cases based on the product's intended use cases, then grows from user feedback and production failures.
2
Multi-turn simulations can use persistent personas to test whether an agent reveals restricted information, breaks guardrails, or gives inappropriate discounts.
3
A useful evaluation process combines end-to-end user testing with granular checks for tool calls, intermediate steps, errors, and the quality of the evaluation itself.
Summary
Chiara Caratelli explains that agents need evaluation because quick prototypes often fail when exposed to many users and unfamiliar conversations. Her process begins with curated examples from product requirements and team discussions. Those examples cover important flows and can run in CI/CD like software tests. The team then adds multi-turn simulations, where one agent impersonates a persistent user trying to break another agent's guardrails. Production feedback and error analysis add real failures to the evaluation set. Caratelli compares prompt changes with model releases: run them against a trusted set first, then use staged rollout or A/B testing. She also separates the user's end-to-end experience from granular checks of tools, workflows, and chained API calls. Evaluation data must be easy for product managers and other nontechnical people to inspect. LLM judges can help, but they need instructions to find errors and must be checked by people. Caratelli is direct that real users and dogfooding remain necessary because teams cannot predict every failure in advance.
Agent evaluation begins with curated examples before production data exists
Caratelli treats prompt and agent work as a data science task because small prompt changes can alter the output. At the start of a project, there is usually no production data, so the team creates a curated set of examples from product manager insights, team discussions, and specific use cases. The first set covers flows the team expects to work and runs like software tests in CI/CD. This gives the team a way to see what breaks, fix it, and iterate quickly. It also makes the main use cases explicit before the agent reaches users.
Multi-turn personas expose guardrail failures that single-turn tests miss
Single-turn conversations do not reveal whether a persistent user can gradually circumvent guardrails. Caratelli describes creating an agent persona with a defined intention, such as obtaining information that should remain private or persuading the system to give an inappropriate discount. The simulated user keeps trying across several turns, even after the target agent refuses. The team used short conversations of roughly four or five turns. When a simulation succeeded in extracting information, that failure became something to fix and test again.
Fixes should start simply and become more involved when the failure demands it
The response to an evaluation failure depends on the type of problem. Some issues can be addressed with a prompt change. Others require changes in application code, especially when the issue is a security problem. Caratelli also mentions adding a reviewer that checks the output through a content moderation step before the response reaches the user. Her advice is to start with the simplest intervention and add more machinery only when that does not solve the problem. The evaluation process therefore connects directly to the kind of fix the team chooses.
Caratelli describes a progression from examples created by the team, to feedback from internal users, to production data. Once the system has real conversations, error analysis becomes a major part of evaluation. If a user manages to extract information from the agent, that case is added to the evaluation set. The team then runs it whenever the code changes, or periodically according to how often the system is updated. This turns a discovered failure into a regression test rather than leaving it as an isolated incident.
Prompt changes can use the same rollout discipline as model changes
A prompt update changes the behavior of the production system in much the same way that a new model does. Caratelli recommends comparing the new prompt with the previous version on a trusted evaluation set. The team checks whether the new version improves results, whether existing tests now fail, and whether it introduces new errors. After that offline comparison, the change can be rolled out gradually, including through an A/B test. These practices come from traditional data science and machine learning, but Caratelli applies them to agents because agents are now being released into production.
End-to-end testing must be paired with checks inside the agent workflow
An agent combines a prompt and model with tools, workflows, and sometimes other agents. Tool calls can trigger whole workflows, while chained API calls add more opportunities for a wrong result. Caratelli says that testing only the final response does not explain where a failure occurred. Teams need an end-to-end view of the user's experience and more granular checks for individual steps, similar to unit tests in software engineering. For example, a food-search agent should return food that matches a gluten-free request, while developers also inspect the search results and tool calls that produced the response.
Evaluation data has to be readable by product and domain experts
Traditional evaluation platforms may not show complex agent workflows in a way that helps everyone involved. Caratelli's team built applications to visualize conversations, tools, and domain-specific activity so nontechnical people could judge whether an interaction was good enough. Product managers need to inspect this information because they guide the product itself. She also recommends looking at error distributions and allowing free-form summaries of what went wrong before imposing a fixed taxonomy. Teams can later group those summaries into categories such as tool errors or timeouts, while also using traces, logs, and evaluation results.
LLM judges need human oversight and direct testing of the actual app
An LLM judge can assess questions such as whether a response is good or whether its tone is appropriate, but Caratelli warns that judges may be too willing to say everything is fine. They must be instructed to search for errors, and people from both technical and product teams need to review the quality of the evaluation output. She also says that nothing replaces using the application itself. The team should dogfood the complete experience, including the interface, and ask people with different perspectives to try it. A technically acceptable agent can still create a poor user experience.
Release decisions depend on the cost of each error
Caratelli rejects a universal coverage threshold for releasing an agent. Without production data, teams are estimating how the system will behave, and real users are needed to reveal failures. The right tolerance depends on the product. Search can involve precision and recall tradeoffs, but food information may require high precision when allergies or dietary restrictions are involved. A wrong answer about whether food is vegan can be much more serious than a conversational stumble that the agent corrects later. Teams should define acceptable metrics and error levels with engineering and product before release, then use user feedback to improve them.
