Real-time voice agents add problems that text agents largely avoid, including turn detection, interruptions, accents, language switching, latency, and speech recognition errors.
2
Paul van der Boor describes iFood tests with Brazilian delivery riders, where traffic noise, colloquial Portuguese, live data, memory, and safe hands-free use made a laboratory demo much harder to deploy.
3
Floris Fok recommends testing voice agents with randomized synthetic conversations, keeping real-time tools fast, redesigning flows to avoid spelling errors, and measuring where users abandon calls.
Summary
Demetrios Brinkmann, Paul van der Boor, and Floris Fok discuss why voice agents behave differently from text agents. Paul describes the move from separate speech recognition, language model, and text-to-speech pipelines toward speech-to-speech systems, along with iFood experiments involving riders in Brazil. Real-world conditions exposed problems with context windows, hallucinations, language switching, accents, noise, and tool use. Floris explains that real-time APIs are asynchronous systems built around streamed events, interruptions, and turn detection. A pause can mean that someone is thinking or that they have finished speaking, and the right setting differs by person. The guests also cover healthcare assistance, proactive agents, open-source text-to-speech, and voice analytics. Their practical advice is direct: test many speaking styles, give callers feedback while tools run, move slow work into background jobs, design workflows around recognition limits, and inspect call data for repeated failure points.
Real-time voice turns a simple text pipeline into a live interaction problem
Paul van der Boor explains that earlier voice systems took text, generated audio, and made the user wait. Faster inference from Groq made a pipeline of voice transcription, a language model, and text-to-speech feel real time. OpenAI's real-time Voice API and similar systems then enabled streaming voice in and voice out. The difference matters because voice is used while people are moving, driving, or dealing with background noise. A food delivery rider in São Paulo may speak Brazilian Portuguese while traffic is loud and an order needs immediate attention. That setting tests the whole system rather than just the quality of a demo.
Paul says a production voice system combines transcription, speaker detection, language detection, translation, voice generation, and sometimes voice cloning. Prosus evaluates transcription across languages because its products operate globally. The audio signal also contains accent, emotion, and intonation. Those signals may help an agent understand whether someone is annoyed, whether a learner is engaged, or whether a customer needs a faster response. Paul is clear that receiving voice and producing a correct answer is only the start. The system must decide how to use information that text chat does not provide.
Voice models bring back hallucination and language-switching problems
Paul says their real-time tests exposed limits that seemed less visible in ordinary text-to-text interactions. Putting the full conversation into the context window creates its own limits. The model could switch from Portuguese to English after hearing an English term and fail to return to Portuguese, despite the system prompt. It was also more prone to making things up. Paul expects function calling to help because agents will be able to retrieve live information and use tools instead of guessing. Instruction following still depends on the use case, so higher-fidelity conversations require improvements beyond the voice interface itself.
Floris Fok explains that a real-time session is asynchronous. Messages, audio, and events move through a web socket instead of arriving as one user message followed by one complete answer. The API can record an interruption and trim the model's unfinished output, so the model does not assume that the caller heard the rest of a sentence. Turn detection is especially difficult because natural pauses can mean either thinking or finishing. Someone who speaks slowly may trigger a response during a short pause, while someone else may need a different silence threshold. Floris expects adaptive, possibly open-source turn detection to learn a person's speaking pattern.
Making a voice agent slightly less capable can improve the call
Demetrios Brinkmann shares an observation from another builder: an agent that sounds too capable can make users expect human-level performance. When it makes mistakes, callers may become impatient. A less polished agent can lead people to slow down and pronounce words more clearly, which may produce better results for the product. Floris connects this to the way people mirror each other's speaking style. The point is practical rather than cosmetic. Voice quality changes caller behavior, and that behavior affects recognition, turn detection, and the success of the workflow.
Automated voice evaluations can expose failures that scripted tests miss
Floris describes building an evaluation system because the real-time API was new and did not yet have a testing framework. They used the API to test itself, generating many calls with randomized character settings and speaking styles. The tests varied speed, accents, emotion, and language while an agent completed a task such as ordering a pizza. The experiments found that Italian and Spanish could be confused, German accents could cause the agent to respond in German, and slow speech could trigger turn detection inconsistently. Synthetic conversations let the team explore unusual cases repeatedly instead of waiting for them to appear in production.
Voice workflows need fast tools and different interaction patterns
Floris recalls that his first voice agent became silent while it waited for a tool result. The caller could not tell whether the system was still working. He recommends prompting the agent to say that it is calling a tool and to explain what it is doing, while also keeping real-time tools quick. Slow operations should become background jobs with a later message or status update. He also recommends redesigning flows that depend on spelling. Rather than asking the model to transcribe an unfamiliar name directly, the system can offer a short list of possible names and map the spoken answer to a known option.
Call analytics should show where conversations break
Demetrios describes Canonical AI's product analytics for voice agents, which helps builders inspect where users disengage or where a flow takes too long. Floris agrees that logs revealed an overconfident behavior: the agent would guess after a mispronunciation instead of asking the caller to repeat it. They compare live voice analytics to software monitoring with Datadog or Prometheus. A call center should know if many calls hang up after the same step or if a process that should take a minute is ending after only a few seconds. These signals give developers a way to change the flow based on actual conversations.
"For voice, it didn't behave the same way. Of course, we're testing different languages. So, you start with Portuguese, and the moment you mention an English term, it starts responding in English, and it doesn't go back to Portuguese anymore."Paul van der Boor07:54
Who should watch
You are building a phone or voice interface and need to understand why a text-agent prompt cannot simply be copied into a real-time voice API.
Your agent must work with accents, multiple languages, interruptions, background noise, or live business data, and you want concrete testing and flow-design advice.
You are setting up production monitoring for voice calls and need examples of the failures worth measuring, including silence, spelling, overconfidence, and user abandonment.
