Voice agents need application-level controls for latency, transcription mistakes, reliability, response phrasing, synthesis quality, and numerical accuracy.
2
Speech-to-speech systems can preserve vocal context such as tone and sentiment, but they need observable components so engineers can debug and control them.
3
The quality of a voice agent should be judged by customer outcomes, since improvements such as lower latency do not always matter after a certain point.
Summary
Anoop Dawar and Ajeet Grewal explain why production voice agents require much more than a language model connected to speech APIs. Voice systems must handle interruptions, pauses, background voices, latency across several pipeline stages, transcription errors that change meaning, unnatural phrasing, inconsistent synthesis, and unreliable providers. Ajeet describes the engineering work Sierra does at the application layer, including provider switching and monitoring. Anoop then explains why speech-to-speech models are attractive: converting speech into text loses information about tone, sentiment, and the many ways people say the same word. Their proposed direction keeps speech-to-speech systems observable and composable, with separate heads or interpretable stages that let teams inspect what went wrong. The speakers also discuss multilingual speech, synthetic data, accents, and the limits of word error rate as a measure of quality.
Production voice agents break in several places at once
Ajeet Grewal compares voice agents with the early web: prototypes are easy to create, while systems that work reliably at scale are difficult. Voice adds problems that do not appear in chat. There is no clean boundary between the user stopping and the agent speaking, so interruption handling has to distinguish a pause from the end of a thought. The system also has to prevent hallucinations, protect personally identifiable information, follow workflow steps, and connect securely to customer APIs. Ajeet says the application layer has to coordinate all of this rather than treating the language model as the whole product.
Voice latency comes from a chain of dependent stages
A voice response moves through speech recognition, language-model processing, and speech synthesis, so each stage adds delay. Measuring that delay during an ongoing audio stream is itself difficult. Sierra uses pipelining and sometimes makes throwaway language-model calls when it believes a speech segment has ended. Those calls can be wasted if the speaker continues, but they may reduce the wait for the next step. Progress phrases such as "Thank you for asking this question. Let me look that up for you" give the agent time to perform latency-sensitive work without leaving the user in silence.
Word error rate misses mistakes that change the task
Ajeet argues that word error rate is a poor standalone measure for voice agents. If a user says "I don't want to sign up" and the system drops "don't," the meaning and the resulting workflow change completely. That one word matters more than several harmless transcription errors. He separates these major mistakes from minor errors that a language model can often recover from. The application layer also has to monitor provider reliability and switch providers when one is having problems. The useful measure is the effect on customer outcomes, such as whether a business can contain a request successfully.
Spoken responses need different writing and synthesis choices
Text conventions do not transfer directly into speech. URLs, numbered lists, long product descriptions, and detailed prices can sound unnatural when read aloud. Ajeet says voice responses need brevity and phrasing that resembles ordinary human speech. Teams can cache common sentences such as "Got it" or "Thank you" to reduce synthesis latency, but cached audio can create a noticeable shift in tone between sentences. Sometimes the system should accept extra latency to keep the voice consistent. Numbers create another failure mode, since providers may pronounce confirmation codes incorrectly unless the system handles their spacing carefully.
Speech-to-speech can preserve emotion that text removes
Anoop Dawar explains that a transcription such as "hello" captures very little about how the word was spoken. A tired hello, an unwilling hello, and an excited greeting can all become the same text. The speech-to-text, language-model, and text-to-speech pipeline therefore loses information about tone and sentiment. Speech-to-speech models can carry more of that vocal context through the system, allowing the model to respond to the customer's sentiment and keep the generated speech consistent. Anoop says this added richness also makes the system harder to inspect unless observability remains part of the design.
Composable speech systems preserve control while reducing delay
Anoop describes Deepgram's Neuroplex research as a speech-to-speech design that separates the automatic speech recognition and language-model heads while passing latent representations through the system. Removing those intermediate penalties can reduce latency and preserve more of the speech signal's context. Keeping the heads available gives engineers near-real-time information about what happened at each stage. That matters when a response fails, because a team can investigate whether the system misheard the user, reasoned incorrectly, or synthesized the answer poorly. Anoop prefers a composable architecture where teams can replace models and retain control rather than debug an opaque monolith.
Voice quality depends on data, context, and measurable outcomes
The speakers say multilingual voice agents need accurate transcription, accurate synthesis, and language-model responses within the same conversation. Switching languages should not unexpectedly change the speaker's voice or accent. Synthetic data can help, but Ajeet says it needs a baseline model that already synthesizes the target language well. Anoop adds that synthetic data needs a foundation of high-quality real data, or it can produce very poor results. Accent handling also requires enough representative data and training, along with guardrails that prevent jarring accent changes when a speaker moves between languages.
"There is that piece of knowledge that gets lost when you take a speech-to-text system, convert it to text, then you go to an LLM and then you take the LLM output and you go to text-to-speech."Anoop Dawar19:08
Who should watch
You are building a voice agent that works in a demo but becomes unreliable when it reaches real customers or higher call volume.
Your team is deciding whether to use a modular speech pipeline or move toward speech-to-speech models.
You need practical measures for transcription quality, latency, interruption handling, multilingual speech, or provider failures.
