# Building Conversational AI Agents with Voice Michelle Chan, Deepgram | AI in Production 2024 | 11:55 Source: https://www.youtube.com/watch?v=hV1cAAVI4fY Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/building-conversational-ai-agents-with-voice Published: 2024-04-25 Tags: latency, prompt-engineering, voice ## TL;DR - A voice agent needs low latency across speech recognition, language-model generation, and speech synthesis, with 300 milliseconds presented as a human conversation benchmark. - End-of-turn detection depends on tone, context, and whether the person sounds ready to continue, so builders need more than text-only logic. - Natural voice requires attention to prosody, voice identity, filler words, pauses, chunking, and prompts that shape language-model output. ## Summary Michelle Chan explains how to build voice agents that feel more like conversations between people. She describes the basic pipeline: speech is transcribed, sent to a language model, and converted back into speech. The hard parts include keeping latency low, detecting when a person has finished speaking, choosing a voice with suitable prosody, and making generated speech sound natural. She discusses tone, pitch, rhythm, pauses, filler words, breathing, slang, and the way text is chunked before it reaches speech synthesis. The talk also covers practical decisions such as voice branding, reference voices, accents, demographics, and prompt design. In the closing section, Chan introduces Deepgram's Aura text-to-speech product, which was targeting latency under 250 milliseconds for conversational voice. During the questions, she adds that slang and colloquial speech depend on prompts, accents, and the underlying data. ## Key ideas ### Voice agents connect speech recognition, an LLM, and speech synthesis [01:23](https://www.youtube.com/watch?v=hV1cAAVI4fY&t=83s) Michelle describes a voice agent as a pipeline. A customer speaks on a phone, the system transcribes that speech into text, sends the text to an LLM, and turns the generated response back into speech. She names appointment booking, customer support, outbound sales, and interviews as examples. A support call might begin with the AI collecting information during triage, then pass the case to a human agent. ### Latency changes whether a voice interaction feels natural [02:42](https://www.youtube.com/watch?v=hV1cAAVI4fY&t=162s) Michelle says latency is one of the first issues builders need to address. She cites research suggesting that around 300 milliseconds is a useful benchmark for a conversational exchange before people begin to experience the interaction as slow or unnatural. Adding an LLM makes that target difficult because the system must recognize speech, generate an answer, and synthesize audio. She presents this as a target for voice systems rather than a problem that has already been solved. ### End-of-turn detection depends on how a person speaks [04:04](https://www.youtube.com/watch?v=hV1cAAVI4fY&t=244s) A voice agent has to decide when the user has finished talking before it responds. Michelle says this is nuanced because the answer depends on the person's tone, the context, and the last sentence they were saying. A pause may mean the person is thinking, or it may mean they are finished. She calls this problem endpointing and says Deepgram speech-to-text includes endpointing that builders can use as part of transcription. ### Prosody makes a synthetic voice sound conversational [04:57](https://www.youtube.com/watch?v=hV1cAAVI4fY&t=297s) Michelle uses prosody to describe the naturalness of speech, including rhythm, pitch, intonation, and pauses. Text-to-speech voices are often tuned for particular settings such as movie narration, news articles, or advertisements. A conversational agent needs a voice designed for dialogue. She recommends defining voice-branding guidelines, collecting reference voices, and specifying tone, emotion, accent, and the demographic the voice should fit. ### Filler words and breathing can provide human-like thinking time [06:32](https://www.youtube.com/watch?v=hV1cAAVI4fY&t=392s) Human conversation includes sounds such as "um" and "uh," breaths, and pauses in appropriate places. Michelle says users experimenting with voice agents have asked an LLM to speak as if it were having a conversation. Builders can also add pauses, including through punctuation such as three dots, and can test filler words. These details differ from the style used when a text-to-speech system reads a news article. ### Chunking controls both response speed and speech quality [07:49](https://www.youtube.com/watch?v=hV1cAAVI4fY&t=469s) LLMs generate output word by word, so builders need to decide how to group words and sentences before sending them to speech synthesis. Some vendors provide real-time input streaming. A team can also create its own chunking logic based on its conversational data. Sending output one word at a time may reduce waiting, but it can make the audio sound chopped up. The right choice depends on the model and the desired balance between latency and voice quality. ### Prompts alone do not automatically make LLM output sound spoken [08:37](https://www.youtube.com/watch?v=hV1cAAVI4fY&t=517s) Michelle says language-model outputs do not sound conversational by default. Builders can experiment with prompts, output structure, pauses, and other controls before passing text to speech synthesis. The goal is to make the generated text sound more natural when spoken aloud. This means treating the language model's output as something that needs shaping, rather than assuming ordinary generated prose will work directly in a voice interaction. ### Slang needs language, accent, and data to work together [10:18](https://www.youtube.com/watch?v=hV1cAAVI4fY&t=618s) In the discussion about slang, Michelle says builders can change the words an agent uses, and prompts can help make the output more conversational. She also says the technology is still developing. Slang sounds believable only when it fits with the accent and the rest of the speech, so the issue is not limited to swapping in individual terms. The data set also affects how well an agent can produce colloquial language. ## Notable quotes - "The first problem is, how do we know when the human stops talking?" (04:04) - "Ideally in a conversational conversation, a 300 millisecond latency is the benchmark we can look into." (03:09) - "What makes human conversations natural is things like um, I'm also having a lot of ums right now, and breaths like thinking time." (06:32) - "LLM outputs by default don't sound like a conversation." (08:37) - "It has to sound real when it's in the accent together with a slang." (11:18) ## Tools & references mentioned - Deepgram - Aura - Spotify ## Who should watch - You are building a phone-based agent and need to reason about latency, endpointing, and when the user has finished speaking. - Your system already produces text, but its audio sounds like narration rather than a live conversation. - You are choosing a synthetic voice and need practical guidance on prosody, pauses, filler words, accents, and conversational style. ## Related talks - [Voice Agent Use Cases](https://mlopstalks.com/talks/voice-agent-use-cases) (Anurag Beniwal, ElevenLabs, 51:05) - [The Challenge with AI Voice Agents](https://mlopstalks.com/talks/the-challenge-with-ai-voice-agents) (Paul van der Boor & Floris Fok, Prosus Group, 47:38) - [Building Real-Time, Reliable Voice AI: From Simulation to Production](https://mlopstalks.com/talks/building-real-time-reliable-voice-ai-from-simulation-to-production) (Brooke Hopkins, Coval, 57:13) - [Voice AI's Biggest Weakness Exposed](https://mlopstalks.com/talks/voice-ais-biggest-weakness-exposed) (Brooke Hopkins, Coval, 38:22) - [I Built A Trustworthy Voice Assistant](https://mlopstalks.com/talks/i-built-a-trustworthy-voice-assistant) (, 12:17)