Production voice agents need a balance between the control of cascaded systems and the natural interaction of speech-to-speech models.
2
Voice agents can mask slow tools, APIs, and retrieval by using a smaller foreground model while a more capable model works in the background.
3
Customer support agents currently reach about Level 1 performance, while learning expert judgment from Level 2 agents remains an open problem.
Summary
Anurag Beniwal describes the engineering choices behind production voice agents. He prefers architectures between simple cascaded pipelines and fully speech-to-speech systems because enterprises need control over models, tools, latency, and failures. A smaller model can keep a caller engaged while retrieval, tool calls, or a more expensive model work in the background. He explains why voice adds problems that chat already has, including transcription errors, background noise, accents, multi-speaker audio, and turn-taking. In customer support, current agents can handle routine Level 1 work, but they struggle to reproduce the judgment of expert Level 2 agents. Anurag also discusses inbound sales, booking, reservations, home services, and concierge tasks as practical use cases. He argues that voice platforms need different interfaces for business operators, developers, and enterprise teams, with fewer technical controls for non-engineers and more architectural choices for developers.
Voice platforms need the right amount of control for each user
Anurag says voice systems create a tension between pre-configuration and flexibility. More controls let developers tune buffer sizes, latency, and speech quality, but users who do not understand those settings can easily produce a bad experience. Customer support managers often own the operation while having fewer engineers available. Anurag describes interfaces that let non-technical authors define agent behavior through familiar standard operating procedures, or SOPs. They can write detailed procedures, then give natural-language feedback when an agent gives wrong information, breaks a rule, or gets confused by an unclear procedure. The platform must turn that feedback into changes to the agent specification.
Voice adds transcription and turn-taking failures to the problems already found in chat
Anurag says multi-turn chat already struggles with recovery, clarification, changing intent, and knowing when to ask a follow-up question. Voice adds background noise, multiple speakers, accents, and different speaking patterns. Accurate automatic speech recognition matters especially for names, email addresses, and order numbers because one transcription error can break the whole interaction. Turn-taking is also still open. A neural model can use speech and text, while acoustic features such as pitch and RMS energy can detect with high confidence that a person has stopped speaking. Anurag favors a hybrid approach in which fast acoustic signals can reduce latency and neural models handle harder cases. He says internal benchmarks from real enterprise data are often more useful than public benchmarks.
A constellation of models helps voice agents stay responsive
Voice callers tolerate silence poorly, so using a large model for every turn can make an agent feel broken. Anurag describes a foreground model that handles simple conversation and keeps the caller engaged while a background model performs deep retrieval, research, or a slow API call. The agent might give a quick answer, ask whether the caller wants more detail, and then use the deeper result when it is ready. Different models can also handle different tasks. Anurag says Claude Haiku worked well for tool calling, while a smaller fine-tuned model gave more control over response generation in Amazon customer support. He has rarely seen one model handle every complex production task well.
Model consolidation can create interference between tasks
Anurag explains that production systems used separate models for intent detection, response generation, dialogue state tracking, and action prediction before large language models became common. Teams tried consolidating those tasks into fine-tuned models such as Flan and Mistral, but improvements on one task could hurt another. He compares this to Whac-A-Mole. Earlier systems kept conversations contained by selecting and filling approved response templates from a constrained set. Modern systems can use more capable models, but Anurag still expects a smaller constellation for complex support workloads. For a small or medium business with a simple question-answering flow and a few tool calls, one model may be enough.
Expert customer support judgment is harder to capture than support documentation
A knowledge base plus tools can reach roughly Level 1 support, according to Anurag. Level 2 agents are domain experts who make decisions about exceptions and subtle policy differences. Their judgment often depends on details such as account age, location, loyalty status, or credit usage. Anurag describes retrieving high-quality conversations from expert agents, checking them for the right policy context, and passing useful examples to the main agent. Similar-looking conversations can require opposite actions, so retrieval alone is insufficient. Preference tuning or another trained component may be needed to distinguish the correct example. Capturing the reasoning behind expert decisions is difficult because agents are paid according to call volume, which makes extra documentation costly.
Voice can produce better context because people speak more freely than they type
Demetrios Brinkmann says he gives more context during a voice interaction than in text. Anurag agrees that this is a practical benefit of voice. People can explain their situation, correct the agent, and think aloud instead of compressing everything into a short typed summary. He connects this to support workflows where agents spend substantial time writing follow-up emails. A person can speak the required content, then correct a generated email by voice. Anurag also discusses systems that combine automatic speech recognition with language understanding. These systems can omit irrelevant speech, back-channeling, and hesitation instead of transcribing every word literally. The trade-off is less direct control over the language model, which is why this approach is easier to accept in some personal productivity applications than in tightly controlled enterprise systems.
Inbound qualification and booking have clearer value than cold outbound calling
Anurag lists customer support, inbound sales, outbound sales, booking, reservations, and concierge work as voice-agent uses. Inbound lead qualification is attractive because the customer has already expressed interest. The agent can replace a form and ask whether the lead is a fit. He is more cautious about outbound calls because people may resent discovering that the caller is automated, although some companies are already building outbound sales products. Home services provide a concrete case. Someone with an urgent plumbing problem wants a quote and an available contractor today, rather than waiting for a website form to be reviewed. A voice agent could contact contractors, compare availability and price, and arrange the job. Booking and reservations also fit because an authenticated customer can authorize an agent to make an appointment.
The useful design space sits between simple cascades and speech-to-speech
Anurag argues that fully speech-to-speech systems may be fast and natural, but enterprises still need to fix behavior, change models during an outage, and control specific parts of the stack. Simple cascaded systems provide control but can sound unnatural and expose too many configuration knobs. He describes intermediate patterns, such as fusing ASR with the language model, combining ASR with turn-taking, and using a foreground model alongside a more capable background model. Conversational TTS can receive the current utterance plus the conversation history, which helps preserve tone across longer interactions and avoids an inappropriate laugh or emotional shift. He says developer platforms should make these combinations easier to test, since the trade-offs involve control, quality, latency, and reliability.
"I can have a smaller model for these more cursory conversations, high-level chit chat, or turns that require low intelligence, where these models keep you engaged and then they delegate the more intense task in the background to a more expensive model."Anurag Beniwal17:41
Who should watch
You are building a voice agent and need to choose between a cascaded pipeline, speech-to-speech, or an intermediate architecture.
Your team is working on customer support automation and needs to understand why routine Level 1 work is easier than expert exception handling.
You are designing a voice platform for operators and developers and want examples of where configuration, feedback, retrieval, and model choice create product problems.
