AI agents need simulation-based testing because the same input can produce different outputs and the agent may take many valid paths.
2
A simulation suite uses mock users, the production agent, and a judge agent to test goals, policies, edge cases, and regressions across conversations.
3
Voice simulations test the full stack, including transcription, reasoning, speech synthesis, turn-taking, accents, emotion, speed, and background noise.
Summary
Sachi Shah explains how Sierra tests customer-facing agents before they interact with real users. Traditional unit, integration, and smoke tests still help, but agents are non-deterministic and can reach many valid outcomes. Tests therefore need to assess whether the user achieved their goal, rather than whether the agent followed one prescribed sequence. Sierra creates simulations with detailed mock-user personas, real agent conversations, and a judge agent that scores expected behaviors. These suites cover common paths, missing information, wrong information, and other variations, then run repeatedly as regression tests. Shah also describes voice simulations that inject background noise, accents, languages, speaking speeds, and emotional states while testing the complete voice stack. She recommends generating suites from existing knowledge bases, transcripts, and SOPs, then running selected simulations in CI/CD. The talk is practical about current limits: teams still lack a single agreed metric for simulation value, so they watch production issues, containment, and goal completion.
Sierra treats agents differently from traditional deterministic software because the same input can produce different outputs. An agent may reason and improvise rather than follow fixed steps, especially when it has goals, policies, conversation history, memory, and other context. A return request illustrates the problem. The right outcome might be processing the return, offering a promotion, or solving the speaker problem so the customer no longer needs a return. The test should ask whether the user achieved the goal. Shah says the possible paths can be effectively infinite, so a narrow expected response is not enough.
The agent development lifecycle adds simulations to the release loop
Shah describes an agent development life cycle that starts with defining goals and guardrails, moves into simulations and regression tests, then continues through release, production observation, and optimization. The cycle repeats because teams cannot predict every issue before launch. Simulation results should resemble what happens in the real world, while production behavior reveals more cases to add or improve. This gives teams a place to test major agent changes before release and a way to keep checking quality after deployment.
A simulation combines a mock user, the real agent, and a judge
Each simulation has three parts. The mock user receives a persona, location, language, device, reason for contacting support, and a specific goal. For example, a traveler in another country may have lost a credit card and need an urgent replacement. The mock user then interacts with the actual agent, producing a conversation. A judge agent reads the full transcript, considers the user's reason for calling, and grades whether the expected behaviors happened. Shah recommends splitting separate requirements into separate criteria instead of combining them with conditions such as doing A or B. This gives the judge clearer instructions.
Simulation suites cover smooth paths and messy customer situations
Teams can create hundreds or thousands of simulations and run them whenever the agent changes. An e-commerce suite might include a customer who knows the order number, one who needs an order found through an email address, and one who provides the wrong order number. These cases test how gracefully the agent handles missing or incorrect information. Once defined, the suite becomes a regression source of truth that collaborators can rerun after changes. Shah says this avoids repeating manual tests and helps teams check whether a new version introduced failures.
Voice testing needs realistic conditions across the whole stack
Voice adds emotional state, background noise, accents, speaking speed, transcription, reasoning, and speech synthesis. A customer may be angry after a stolen delivery, while the agent still needs to show empathy and solve the issue efficiently. The simulation can test whether a complex number is transcribed correctly through noise and whether the agent reads a birth date or license number accurately. Voice may also require different authentication or interaction logic from chat. Sierra uses a dual voice-loop design: a simulated user generates audio, noise and speaking instructions modify it, and audio streams through the agent's voice loop until the conversation ends.
CI/CD integration keeps simulation feedback close to development
Sierra generates simulation suites from sources of truth such as knowledge bases, conversation transcripts, and standard operating procedures. Users can then edit the generated cases. The suites can run inside CI/CD through GitHub Actions and command-line commands. Teams can choose simulations that run before a release or before a commit so the main branch stays clean. Shah connects this placement to the balance between development speed and reliability. Running simulations too late slows the development cycle, while running them earlier gives developers a chance to fix failures before release.
Simulation data can support quality checks before production
Shah says simulated conversations can support manual quality assurance as well as automated testing. They can also provide predictive information before an agent talks to real customers. For voice, a team might inspect the latency of every turn and decide whether it needs optimization. In the question period, Shah says there is no single established industry metric for simulation value yet. Sierra watches whether fewer problems appear in real conversations and whether live agents hit customer experience measures such as containment rate and their intended goals.
