Voice agents need low latency and high speech recognition accuracy because errors compound as speech moves through recognition, reasoning, and speech generation.
2
Production voice agents break down around orchestration, observability, interruption detection, turn-taking, and the need to maintain predictable latency during live calls.
3
Deepgram's Flux combines streaming speech recognition, voice activity detection, and end-of-turn detection to handle conversational turn-taking in one system.
Summary
Anoop Dawar explains how voice AI moved from batch transcription toward live systems that combine speech recognition, language-model reasoning, and speech generation. He argues that low latency and accurate recognition are only the starting point. Production systems also need logs, fallbacks, interruption handling, turn detection, and predictable latency for every call. A voice activity detector that treats a short silence as the end of a turn can interrupt people while they are still speaking. Dawar presents Deepgram's Flux as a system that combines streaming speech-to-text, voice activity detection, and end-of-turn detection. He then compares cascaded voice systems with speech-to-speech models. Speech-to-speech systems can preserve information carried by tone and emotion, which is lost when speech is reduced to text. The talk ends with a vision of an audio Turing test, where voice agents understand intent and emotion while responding with natural timing.
Voice AI connects live speech with a reasoning system
Dawar describes Deepgram as a platform that provides speech recognition, speech generation, and voice-agent APIs for developers. Earlier speech-to-text systems mainly processed recorded calls overnight for quality assurance and textual analysis. The shift came when people connected real-time speech recognition and text-to-speech to large language models. That combination made it possible to build agents that listen, reason, and respond during a live business interaction.
Business demand comes from repetitive customer interactions
Dawar says that more than a billion business interactions happen each day and that many involve routine work, such as following rules or answering common questions. He describes call-agent systems as a large labor expense and points to high attrition among people working in that industry. His argument for voice agents starts with automating repetitive interactions, rather than with entertainment or character-based applications.
A voice agent needs both low latency and accurate speech recognition. If the system does not understand business terms, it cannot complete the requested task. Dawar compares the failure to the children's telephone game: errors in the initial speech transcription pass into the reasoning system and then affect the response. He says Deepgram aims to be faster than human processing while maintaining high accuracy.
Dawar says developers can assemble a voice agent quickly by connecting APIs, but the prototype often lacks orchestration and observability. When a call goes wrong, the team needs to inspect the audio, the customer's words, and the language model's actions, then reconstruct the sequence. He also describes production problems around fallbacks and barge-in, where the agent starts speaking before the person has finished.
A basic voice activity detector can assume that a person has finished after a fixed period of silence, such as 400 milliseconds. Dawar says natural speech contains pauses while people think or separate parts of an identifier. A person dictating a seven-digit ID may pause in the middle, so a silence threshold can cause the system to take the turn too early. The agent needs to understand whether a thought is complete, not only whether sound has stopped.
Live voice calls require a different production model
Dawar says voice agents need predictable latency across interactions because a silent response makes users think the system has failed. In a conventional language-model application, tokens can continue arriving while work is distributed across GPUs. A live call keeps processing tied to the interaction, so capacity and failures affect the experience differently. The system must balance the delay needed to avoid interrupting someone against the delay users will tolerate before hearing a response.
Flux combines the components needed for turn-taking
To reduce the interruption and latency trade-off, Dawar says a production system needs streaming speech-to-text, voice activity detection, and end-of-turn detection. Streaming transcription lets the language model start preparing while the person is still speaking. Flux combines all three functions in one conversational speech recognition system. Dawar presents this as a way to detect when a speaker has completed a thought without forcing developers to assemble the underlying modeling components themselves.
Speech-to-speech models can preserve emotion in delivery
Dawar contrasts cascaded systems, which pass speech through transcription, a language model, and text-to-speech, with speech-to-speech models. The same word can carry different meanings through tone, such as excitement or disdain. A text transcript loses much of that information, while a speech-to-speech system can carry emotional cues through the interaction. He says cascaded and speech-to-speech approaches will fit different applications.
Natural voice agents need to understand more than words
Dawar says Deepgram created 15,000 different embeddings for the word 'hello' to illustrate how many ways one word can be expressed. He imagines applying that richness to sentences and conversations so agents can respond to intent and emotion. This could matter in sensitive calls, including situations where a person expresses a risk of self-harm or harm to others. He frames the long-term goal as an audio Turing test that includes accurate recognition, low latency, voice quality, reasoning, interruption handling, and context passing.
