Sierra tests AI agents with repeated simulations, evaluator agents, adversarial prompts, realistic voice conditions, and deterministic access controls.
2
Voice agents need context about interruptions, pauses, accents, latency, and user-specific speaking patterns, while current speech-to-speech systems remain too hallucinatory for many production uses.
3
Sierra's agents use a constellation of models, speculative execution, goals, and guardrails, with code and no-code tools that let business and engineering teams work on the same system.
Summary
Zack Reneau-Wedeen explains why AI agents need different development methods from traditional software. Sierra runs conversations repeatedly, checks them against task-specific criteria, and uses evaluator agents to find failures. Its simulations include background noise, accents, microphone differences, abuse, prompt hijacking, and customer-specific failure modes. Deterministic systems still control permissions, so a model cannot access data beyond what the logged-in user could access. Voice adds harder problems around interruption, pauses, latency, and speech style. Sierra addresses response time with a constellation of models and speculative execution, often starting retrieval before the system knows whether retrieval will be needed. Zack also describes a hybrid code and no-code platform, customer-led evaluation, outcome-based pricing, and the APX program, which trains new graduates across product management and engineering.
AI agents need repeated simulations instead of one-pass tests
Zack says AI systems are expensive, slower as reasoning improves, and non-deterministic, so traditional unit testing is insufficient. Sierra may run the same conversation 5, 10, or 15 times and inspect what changed. A user agent interacts with the product agent, while an evaluator agent checks whether the conversation met a defined checklist. Critical simulations run when a commit is merged to main or a release is scheduled. Sierra handles the cost through parallel execution, so hundreds of simulations do not have to run one after another. The approach is closer to evaluating distributions of possible conversations than checking one fixed output.
Simulation has to include the messy conditions of real phone calls
Sierra applies simulations to both chat and voice. Voice simulations draw from libraries of background noise, voices, and other variables that can make a call difficult. Zack gives the example of SiriusXM customers calling from a car, a busy street, or a setting with poor microphone quality. Accents and different speaking environments also matter. Sierra's team works through these conditions as a product checklist, then repeats them across hundreds of simulations. Adversarial cases are included as well. Some simulate ordinary customer problems, while others abuse the agent, test self-harm or bullying scenarios, or try to expose weaknesses specific to a customer.
Sensitive data access stays under deterministic control
Zack separates model behavior from permission enforcement. Traditional unit tests check that an agent's access controls match what a user would have on the normal website. If a user logs into a service such as Sonos, the model receives access to the data available within that user's account, such as connected speakers, but it cannot use the model to obtain information outside that account. The model can take actions only when the user would have permission to take those actions. Sierra still uses simulations to test whether the agent reveals information it is allowed to access, since deciding what to say remains non-deterministic. Zack describes the overall development process as a first-principles mix of traditional software methods and AI-specific methods.
Voice agents must interpret conversational timing rather than fixed pauses
Zack describes voice as a major opportunity and a demanding engineering problem. An interruption can mean disagreement, encouragement, a request to change direction, or an actual attempt to take the turn. A system that stops whenever it detects speech feels as if it is talking over the caller. Long pauses create a related problem because the system must distinguish hesitation from the end of a turn. People also have different cadences, speeds, and habits. Zack says customer-level settings can account for groups such as older callers or authentication-heavy flows, but fully personal adaptation may require speech-to-speech models that infer directly from audio tokens. Those models remain too hallucinatory for many production environments, so Sierra often relies on modular transcription and text-to-speech pipelines.
New models require customer-specific testing rather than benchmark scores alone
When a new model appears, Sierra runs a broad evaluation suite, then tests different prompts, workflows, and fine-tuning choices over several days or weeks. Zack says the real question is whether the new model has a higher ceiling than the local maximum reached with the previous model. Voice evaluation also needs language and dialect coverage, such as Brazilian Portuguese versus Portugal Portuguese. Sierra works with callers who actually speak the relevant language and can test realistic situations. Zack describes a case where an accented voice caused a system to switch languages unexpectedly. Such behavior may feel natural in some contexts, but it still needs to be understood before the model is used in production.
Agent improvement depends on finding the cause of failure
Zack says task completion metrics and heat maps matter because teams need to know what went wrong before they can decide how to fix it. An agent may lack the right answer, follow an incorrect standard operating procedure, or be asked to reason about too many things at once. Human handoffs provide another source of training data. Sierra can inspect what a human did after a transfer and identify missing knowledge or procedure steps. Its platform can draft knowledge-base articles and prioritize gaps that appear across many conversations, although teams can decide that some information should remain unavailable to the agent. Zack describes this as a way to use production failures to improve the system's knowledge.
A constellation of models and speculative execution keeps voice responses fast
A single response may invoke 10, 15, or 20 models. These can include embedding models for retrieval, frontier models for complex reasoning, and fast classification models for identifying the task. Sierra runs many of them in parallel. Zack gives retrieval-augmented generation as an example: the system can start a knowledge lookup while another model decides whether that lookup is needed. If the lookup turns out to be useful, its result is already available. This speculative execution matters most in voice, where a five-second response and a 1.5-second response feel very different. The architecture was shaped by voice latency requirements, then became the foundation for other channels as well.
Goals and guardrails fit agents better than fixed decision trees
Zack says decision trees and standard operating procedures can still help people describe how an agent should behave. Sierra's architecture is closer to goals and guardrails. A goal might be helping a customer reset a radio, make a payment, or avoid cancelling a subscription. Guardrails define what offers the agent may make, what experience it should provide, and what facts it must state accurately. This structure fits the web of model calls better than a single straight flow. Sierra also builds traceability into the system because developers generally cannot inspect why an LLM made a particular decision. The product needs to show why a decision occurred and let a team fix one problem without creating another.
Code and no-code tools bring business and engineering into the same workflow
Sierra began as a developer platform, but Zack says most agents are now built with no-code tools. The code and no-code versions are isomorphic, so a customer-service team can build a journey and an engineer can replace one part with code when an API needs custom handling. This lets business teams define goals, procedures, and quality expectations while engineers handle integrations and specialized behavior. Zack connects the product design to Sierra's experience working directly with customers. He also describes evaluation sessions where business and technical staff review real conversations together. Business specialists catch problems that engineers might accept as technically reasonable, while engineers can turn those findings into changes to the agent or its tools.
"Most of the time I would say the solutions come at a higher level in the agent architecture or in some of the underlying task prompts."Zack Reneau-Wedeen40:50
Who should watch
You are building a customer-facing AI agent and need a testing approach that covers repeated conversations, adversarial behavior, and real-world voice conditions.
Your voice system struggles with latency, interruptions, pauses, accents, or speech-to-speech hallucinations, and you want to understand the engineering tradeoffs.
Your business and engineering teams disagree about who should own agent behavior, evaluation, and integrations, and you are considering a shared code and no-code workflow.
