# Voice Agent Use Cases

Anurag Beniwal, ElevenLabs | MLOps Podcast | Episode 374 | 51:05
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=no_gxL40NC8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/voice-agent-use-cases
Published: 2026-06-19
Tags: agents, latency, tool-use, voice

## TL;DR
- Production voice agents need a balance between the control of cascaded systems and the natural interaction of speech-to-speech models.
- Voice agents can mask slow tools, APIs, and retrieval by using a smaller foreground model while a more capable model works in the background.
- Customer support agents currently reach about Level 1 performance, while learning expert judgment from Level 2 agents remains an open problem.

## Summary
Anurag Beniwal describes the engineering choices behind production voice agents. He prefers architectures between simple cascaded pipelines and fully speech-to-speech systems because enterprises need control over models, tools, latency, and failures. A smaller model can keep a caller engaged while retrieval, tool calls, or a more expensive model work in the background. He explains why voice adds problems that chat already has, including transcription errors, background noise, accents, multi-speaker audio, and turn-taking. In customer support, current agents can handle routine Level 1 work, but they struggle to reproduce the judgment of expert Level 2 agents. Anurag also discusses inbound sales, booking, reservations, home services, and concierge tasks as practical use cases. He argues that voice platforms need different interfaces for business operators, developers, and enterprise teams, with fewer technical controls for non-engineers and more architectural choices for developers.

## Key ideas
### Voice platforms need the right amount of control for each user
[00:00](https://www.youtube.com/watch?v=no_gxL40NC8&t=0s)
Anurag says voice systems create a tension between pre-configuration and flexibility. More controls let developers tune buffer sizes, latency, and speech quality, but users who do not understand those settings can easily produce a bad experience. Customer support managers often own the operation while having fewer engineers available. Anurag describes interfaces that let non-technical authors define agent behavior through familiar standard operating procedures, or SOPs. They can write detailed procedures, then give natural-language feedback when an agent gives wrong information, breaks a rule, or gets confused by an unclear procedure. The platform must turn that feedback into changes to the agent specification.

### Voice adds transcription and turn-taking failures to the problems already found in chat
[05:21](https://www.youtube.com/watch?v=no_gxL40NC8&t=321s)
Anurag says multi-turn chat already struggles with recovery, clarification, changing intent, and knowing when to ask a follow-up question. Voice adds background noise, multiple speakers, accents, and different speaking patterns. Accurate automatic speech recognition matters especially for names, email addresses, and order numbers because one transcription error can break the whole interaction. Turn-taking is also still open. A neural model can use speech and text, while acoustic features such as pitch and RMS energy can detect with high confidence that a person has stopped speaking. Anurag favors a hybrid approach in which fast acoustic signals can reduce latency and neural models handle harder cases. He says internal benchmarks from real enterprise data are often more useful than public benchmarks.

### A constellation of models helps voice agents stay responsive
[15:20](https://www.youtube.com/watch?v=no_gxL40NC8&t=920s)
Voice callers tolerate silence poorly, so using a large model for every turn can make an agent feel broken. Anurag describes a foreground model that handles simple conversation and keeps the caller engaged while a background model performs deep retrieval, research, or a slow API call. The agent might give a quick answer, ask whether the caller wants more detail, and then use the deeper result when it is ready. Different models can also handle different tasks. Anurag says Claude Haiku worked well for tool calling, while a smaller fine-tuned model gave more control over response generation in Amazon customer support. He has rarely seen one model handle every complex production task well.

### Model consolidation can create interference between tasks
[20:00](https://www.youtube.com/watch?v=no_gxL40NC8&t=1200s)
Anurag explains that production systems used separate models for intent detection, response generation, dialogue state tracking, and action prediction before large language models became common. Teams tried consolidating those tasks into fine-tuned models such as Flan and Mistral, but improvements on one task could hurt another. He compares this to Whac-A-Mole. Earlier systems kept conversations contained by selecting and filling approved response templates from a constrained set. Modern systems can use more capable models, but Anurag still expects a smaller constellation for complex support workloads. For a small or medium business with a simple question-answering flow and a few tool calls, one model may be enough.

### Expert customer support judgment is harder to capture than support documentation
[25:38](https://www.youtube.com/watch?v=no_gxL40NC8&t=1538s)
A knowledge base plus tools can reach roughly Level 1 support, according to Anurag. Level 2 agents are domain experts who make decisions about exceptions and subtle policy differences. Their judgment often depends on details such as account age, location, loyalty status, or credit usage. Anurag describes retrieving high-quality conversations from expert agents, checking them for the right policy context, and passing useful examples to the main agent. Similar-looking conversations can require opposite actions, so retrieval alone is insufficient. Preference tuning or another trained component may be needed to distinguish the correct example. Capturing the reasoning behind expert decisions is difficult because agents are paid according to call volume, which makes extra documentation costly.

### Voice can produce better context because people speak more freely than they type
[31:15](https://www.youtube.com/watch?v=no_gxL40NC8&t=1875s)
Demetrios Brinkmann says he gives more context during a voice interaction than in text. Anurag agrees that this is a practical benefit of voice. People can explain their situation, correct the agent, and think aloud instead of compressing everything into a short typed summary. He connects this to support workflows where agents spend substantial time writing follow-up emails. A person can speak the required content, then correct a generated email by voice. Anurag also discusses systems that combine automatic speech recognition with language understanding. These systems can omit irrelevant speech, back-channeling, and hesitation instead of transcribing every word literally. The trade-off is less direct control over the language model, which is why this approach is easier to accept in some personal productivity applications than in tightly controlled enterprise systems.

### Inbound qualification and booking have clearer value than cold outbound calling
[36:35](https://www.youtube.com/watch?v=no_gxL40NC8&t=2195s)
Anurag lists customer support, inbound sales, outbound sales, booking, reservations, and concierge work as voice-agent uses. Inbound lead qualification is attractive because the customer has already expressed interest. The agent can replace a form and ask whether the lead is a fit. He is more cautious about outbound calls because people may resent discovering that the caller is automated, although some companies are already building outbound sales products. Home services provide a concrete case. Someone with an urgent plumbing problem wants a quote and an available contractor today, rather than waiting for a website form to be reviewed. A voice agent could contact contractors, compare availability and price, and arrange the job. Booking and reservations also fit because an authenticated customer can authorize an agent to make an appointment.

### The useful design space sits between simple cascades and speech-to-speech
[42:21](https://www.youtube.com/watch?v=no_gxL40NC8&t=2541s)
Anurag argues that fully speech-to-speech systems may be fast and natural, but enterprises still need to fix behavior, change models during an outage, and control specific parts of the stack. Simple cascaded systems provide control but can sound unnatural and expose too many configuration knobs. He describes intermediate patterns, such as fusing ASR with the language model, combining ASR with turn-taking, and using a foreground model alongside a more capable background model. Conversational TTS can receive the current utterance plus the conversation history, which helps preserve tone across longer interactions and avoids an inappropriate laugh or emotional shift. He says developer platforms should make these combinations easier to test, since the trade-offs involve control, quality, latency, and reliability.

## Notable quotes
- Anurag Beniwal: "I think there is always a tension between pre-configuration and giving more flexibility." (00:30)
- Anurag Beniwal: "If you get any of that wrong and I don't have mechanisms to correct that or my transcription is not super accurate, then there is no way my dialogue or the conversation will succeed, it just fails." (08:16)
- Anurag Beniwal: "I can have a smaller model for these more cursory conversations, high-level chit chat, or turns that require low intelligence, where these models keep you engaged and then they delegate the more intense task in the background to a more expensive model." (17:41)
- Anurag Beniwal: "I do believe in many production systems I haven't seen one model doing everything, especially for more complex situations." (22:49)
- Anurag Beniwal: "For some time now, this somewhere in between is the right approach, not the old cascaded way, just chaining them." (48:45)

## Tools & references mentioned
- ElevenLabs
- Amazon
- Vapi
- LangChain
- Sierra
- Decagon
- Claude Opus
- Claude Sonnet
- Claude Haiku
- GPT
- Flan
- Mistral
- BERT
- PipeCat
- Smart Turn V2
- WhisperFlow
- Ultravox
- fixie.ai
- OpenAI
- Google Gemini
- Visa
- MLflow

## Who should watch
- You are building a voice agent and need to choose between a cascaded pipeline, speech-to-speech, or an intermediate architecture.
- Your team is working on customer support automation and needs to understand why routine Level 1 work is easier than expert exception handling.
- You are designing a voice platform for operators and developers and want examples of where configuration, feedback, retrieval, and model choice create product problems.

## Related talks

- [Voice AI's Biggest Weakness Exposed](https://mlopstalks.com/talks/voice-ais-biggest-weakness-exposed) (Brooke Hopkins, Coval, 38:22)
- [Building Conversational AI Agents with Voice](https://mlopstalks.com/talks/building-conversational-ai-agents-with-voice) (Michelle Chan, Deepgram, 11:55)
- [Building Real-Time, Reliable Voice AI: From Simulation to Production](https://mlopstalks.com/talks/building-real-time-reliable-voice-ai-from-simulation-to-production) (Brooke Hopkins, Coval, 57:13)
- [The Coming Revolution of AI Agents](https://mlopstalks.com/talks/the-coming-revolution-of-ai-agents) (Jazmia Henry, Iso AI & Rogerio Bonatti, Microsoft & Julia Kroll, Deepgram & Joshua Alphonse, PremAI, 28:59)
- [Real-Time Voice Agents in Production](https://mlopstalks.com/talks/real-time-voice-agents-in-production) (Panos Stravopodis, Elyos AI, 42:16)