Watch on YouTubeA voice agent needs low latency across speech recognition, language-model generation, and speech synthesis, with 300 milliseconds presented as a human conversation benchmark.
End-of-turn detection depends on tone, context, and whether the person sounds ready to continue, so builders need more than text-only logic.
Natural voice requires attention to prosody, voice identity, filler words, pauses, chunking, and prompts that shape language-model output.
Michelle Chan explains how to build voice agents that feel more like conversations between people. She describes the basic pipeline: speech is transcribed, sent to a language model, and converted back into speech. The hard parts include keeping latency low, detecting when a person has finished speaking, choosing a voice with suitable prosody, and making generated speech sound natural. She discusses tone, pitch, rhythm, pauses, filler words, breathing, slang, and the way text is chunked before it reaches speech synthesis. The talk also covers practical decisions such as voice branding, reference voices, accents, demographics, and prompt design. In the closing section, Chan introduces Deepgram's Aura text-to-speech product, which was targeting latency under 250 milliseconds for conversational voice. During the questions, she adds that slang and colloquial speech depend on prompts, accents, and the underlying data.
Michelle describes a voice agent as a pipeline. A customer speaks on a phone, the system transcribes that speech into text, sends the text to an LLM, and turns the generated response back into speech. She names appointment booking, customer support, outbound sales, and interviews as examples. A support call might begin with the AI collecting information during triage, then pass the case to a human agent.
Michelle says latency is one of the first issues builders need to address. She cites research suggesting that around 300 milliseconds is a useful benchmark for a conversational exchange before people begin to experience the interaction as slow or unnatural. Adding an LLM makes that target difficult because the system must recognize speech, generate an answer, and synthesize audio. She presents this as a target for voice systems rather than a problem that has already been solved.
A voice agent has to decide when the user has finished talking before it responds. Michelle says this is nuanced because the answer depends on the person's tone, the context, and the last sentence they were saying. A pause may mean the person is thinking, or it may mean they are finished. She calls this problem endpointing and says Deepgram speech-to-text includes endpointing that builders can use as part of transcription.
Michelle uses prosody to describe the naturalness of speech, including rhythm, pitch, intonation, and pauses. Text-to-speech voices are often tuned for particular settings such as movie narration, news articles, or advertisements. A conversational agent needs a voice designed for dialogue. She recommends defining voice-branding guidelines, collecting reference voices, and specifying tone, emotion, accent, and the demographic the voice should fit.
Human conversation includes sounds such as "um" and "uh," breaths, and pauses in appropriate places. Michelle says users experimenting with voice agents have asked an LLM to speak as if it were having a conversation. Builders can also add pauses, including through punctuation such as three dots, and can test filler words. These details differ from the style used when a text-to-speech system reads a news article.
LLMs generate output word by word, so builders need to decide how to group words and sentences before sending them to speech synthesis. Some vendors provide real-time input streaming. A team can also create its own chunking logic based on its conversational data. Sending output one word at a time may reduce waiting, but it can make the audio sound chopped up. The right choice depends on the model and the desired balance between latency and voice quality.
Michelle says language-model outputs do not sound conversational by default. Builders can experiment with prompts, output structure, pauses, and other controls before passing text to speech synthesis. The goal is to make the generated text sound more natural when spoken aloud. This means treating the language model's output as something that needs shaping, rather than assuming ordinary generated prose will work directly in a voice interaction.
In the discussion about slang, Michelle says builders can change the words an agent uses, and prompts can help make the output more conversational. She also says the technology is still developing. Slang sounds believable only when it fits with the accent and the rest of the speech, so the issue is not limited to swapping in individual terms. The data set also affects how well an agent can produce colloquial language.
"What makes human conversations natural is things like um, I'm also having a lot of ums right now, and breaths like thinking time."06:32