# Building Real-Time, Reliable Voice AI: From Simulation to Production

Brooke Hopkins, Coval | MLOps Community | 57:13

Source: https://www.youtube.com/watch?v=AlxvTIou4wg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-real-time-reliable-voice-ai-from-simulation-to-production
Published: 2025-08-29
Tags: evals, guardrails, observability, voice

## TL;DR
- Speech-to-speech reduces latency and preserves audio signals such as tone, background sounds, and conversational context, while cascading systems provide more modular control.
- Voice AI reliability depends on turn detection, asynchronous guardrails, production observability, and evaluations that measure whether conversations achieve their goals.
- Real-time voice systems work best when teams combine models and tools according to the application, including reasoning models for high-risk actions and MCP tools for interface control.

## Summary
Brooke Hopkins and Peter Bakkum compare cascading voice systems with speech-to-speech systems and explain where each fits. Speech-to-speech provides native audio input and output, lower latency, and better access to signals such as tone and background speech. Cascading architectures remain useful when teams need modular classifiers, complex instruction following, or strict control over every step. They discuss semantic VAD as a way to decide whether a speaker has finished, since interruption and turn-taking remain difficult across applications. For production reliability, they recommend asynchronous guardrails, output classifiers, conversation-level checks, and evaluations built around concrete goals and failure modes. The conversation also covers prompt design, tool calling, model mixing, multimodal evaluation, and MCP. Both speakers argue that voice applications should be designed around tasks that benefit from natural conversation rather than simply copying existing IVR flows.

## Key ideas
### Speech-to-speech trades modularity for lower latency and richer audio context
[02:09](https://www.youtube.com/watch?v=AlxvTIou4wg&t=129s)
Peter Bakkum describes speech-to-speech as native audio in and native audio out. Cascading systems let teams insert classifiers and select separate models, but speech-to-speech generally reduces latency and understands audio directly. He points to language switching as a difficult case for cascading systems because teams often choose models for particular language properties. Brooke Hopkins adds that audio carries signals that text loses, including tone, background speakers, and context that can help correct transcription errors. Peter illustrates this with a model noticing that Brooke's daughter sounded happy in the background.

### The right architecture depends on the application's tolerance for latency and mistakes
[08:38](https://www.youtube.com/watch?v=AlxvTIou4wg&t=518s)
Brooke says real-time voice is a strong fit for coaches, AI tutors, and interview assistants, where latency, naturalness, and conversational awareness matter. Cascading systems remain useful for complex customer-service workflows where getting every step right matters more than conversational speed. Peter describes customers using mixed designs, such as real-time for turn detection and the language model while sending output to a separate text-to-speech provider. He also describes using the real-time API only for transcription and semantic turn detection.

### Turn detection remains one of the hardest production problems
[13:13](https://www.youtube.com/watch?v=AlxvTIou4wg&t=793s)
The speakers explain that turn-taking varies by use case. Customer support speech has a different rhythm from language learning, where speakers may pause for a long time while thinking. Semantic VAD classifies whether a person appears finished, using signals such as emphasis, a question ending, or trailing off with an 'um'. It then changes how long the system waits. Brooke connects this to interruption metrics, since a voice agent that cuts people off repeatedly will feel unnatural even when its answers are good.

### Guardrails should run asynchronously instead of blocking every response
[16:51](https://www.youtube.com/watch?v=AlxvTIou4wg&t=1011s)
Peter recommends putting guardrails on the text output from the real-time API, since it arrives before the audio output. A classifier can inspect that text and cut off output when it finds something unacceptable. He says even simple rules that look for specific words can be effective. For longer-running checks, teams can inspect the conversation in the background for policy adherence, system-message compliance, or whether the interaction is going well. The system can then guide the model back toward the required path.

### Conversation-level checks catch failures that single-response checks miss
[20:33](https://www.youtube.com/watch?v=AlxvTIou4wg&t=1233s)
Brooke describes conversational failures as slow deaths. An agent can enter a loop, repeat itself, misunderstand the user's intent, or miss a workflow step over several turns. She argues that teams should check whether the user is progressing toward a goal, whether required steps were completed, and whether new context changes the situation. Peter says a separate reasoning model can review a completed voice session and judge whether it was good in a way that can support quantitative evaluation and model improvement.

### Prompts should describe goals and state transitions rather than scripts
[21:23](https://www.youtube.com/watch?v=AlxvTIou4wg&t=1283s)
Peter says many of the same prompting practices from text systems still apply to speech-to-speech. Teams can use clearer system messages and divide conversations into states. He warns against writing a long list of instructions that says exactly what to say in every situation. For customer support, a better prompt explains the goal and the information required to complete it, such as collecting an account number, address, and email for a refund. This gives the model room to handle situations that the team did not script in advance.

### Evaluation should target probabilities and concrete failure modes
[29:08](https://www.youtube.com/watch?v=AlxvTIou4wg&t=1748s)
Brooke recommends defining a small set of important scenarios, such as booking, rescheduling, and cancelling appointments, then running them repeatedly to expose edge cases. She says voice systems need probabilistic evaluation because teams often care that a bad event never happens or that a desired behavior becomes more likely. Peter adds that reasoning models are effective evaluators for multi-turn voice sessions. Teams can evaluate transcripts, rendered audio, or the full interaction, and can use multimodal judges for properties such as audio quality and timing.

### Voice-native products should use conversation where it adds value
[34:51](https://www.youtube.com/watch?v=AlxvTIou4wg&t=2091s)
Peter expects people to talk to computers more as models become faster and more capable. He points to language learning, customer support, and interface control as applications with room to grow. He is especially interested in MCP because a voice model can select tools and make an RPC without requiring the user to type a command. Brooke adds that voice can automate one side of fragmented industries such as trucking, logistics, and healthcare, even when the underlying organizations cannot all integrate through APIs.

## Notable quotes
- "Speech to speech is very much native audio in native audio out." (03:32)
- "Turn detection is still one of the hardest parts of this architecture." (14:31)
- "Guard rails should be asynchronous and you should not block the conversation while you're running your guardrails." (17:31)
- "You can get much higher levels of autonomy by sharing more about what you're trying to do, the context." (49:40)
- "The really core thing in my opinion is exposing a list of tools to the model." (36:54)

## Tools & references mentioned
- Coval
- OpenAI
- Waymo
- Perplexity
- GPT-4o
- MCP
- Notion
- WebRTC
- Pipecat
- ChatGPT
- Match Group

## Who should watch
- You are choosing between a cascading voice pipeline and a speech-to-speech system and need to understand the practical trade-offs.
- Your voice agent works in production and you need better methods for turn detection, interruption handling, guardrails, or conversation-level evaluation.
- You are designing a voice-native product and want to combine real-time models with external voices, reasoning models, or MCP tools.

## Related talks

- [Real-Time Voice Agents in Production](https://mlopstalks.com/talks/real-time-voice-agents-in-production) (Panos Stravopodis, Elyos AI, 42:16)
- [Voice Agent Use Cases](https://mlopstalks.com/talks/voice-agent-use-cases) (Anurag Beniwal, ElevenLabs, 51:05)
- [The Challenge with AI Voice Agents](https://mlopstalks.com/talks/the-challenge-with-ai-voice-agents) (Paul van der Boor & Floris Fok, Prosus Group, 47:38)
- [Building Conversational AI Agents with Voice](https://mlopstalks.com/talks/building-conversational-ai-agents-with-voice) (Michelle Chan, Deepgram, 11:55)
- [Simulate to Scale: How realistic simulations power reliable agents in production](https://mlopstalks.com/talks/simulate-to-scale-how-realistic-simulations-power-reliable-agents-in-production) (Sachi Shah, Sierra, 21:10)