# Fast, Trustworthy, Reliable Voice Agents: MLOps That Blend LLM Annotation with Human QA Erik Goron, HappyRobot | MLOps Community | 17:08 Source: https://www.youtube.com/watch?v=7lNZONiZqqw Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/fast-trustworthy-reliable-voice-agents-mlops-that-blend-llm-annotation-with Published: 2025-08-27 Tags: evals, fine-tuning, human-in-the-loop, monitoring ## TL;DR - HappyRobot evaluates voice agents with business-level north-star metrics alongside metrics for individual models. - LLMs can generate annotations at scale, while targeted human review aligns those annotations with how people judge conversations. - Production data can be cleaned and used to fine-tune specialized models for different voice-agent use cases. ## Summary Erik Goron describes how HappyRobot moved from manually tuning voice-agent behavior to a production MLOps approach. The system tracks north-star metrics such as user frustration, then uses targeted metrics such as word error rate and turn-detector accuracy to find the models affecting those outcomes. Interruptions are a central example because they depend on turn detection, voice activity detection, latency, conversation context, and the use case. HappyRobot uses LLMs to label transcripts and audio, then compares those labels with human annotations and adjusts prompts to improve agreement. The resulting data supports monitoring and model improvement across use cases. Goron also explains why cascading speech systems remain useful: they cost less, expose each component for monitoring, and currently support better tool calling than speech-to-speech systems. He expects a gradual move toward speech-to-speech once those systems provide enough quality and observability. ## Key ideas ### HappyRobot shifted from conversational feel to production metrics [00:00](https://www.youtube.com/watch?v=7lNZONiZqqw&t=0s) HappyRobot's early voice platform was improved through "vibe metrics", meaning the team judged how conversations felt and added practical hacks. As the platform grew to almost a million minutes of calls, the team could measure more of the system. Goron says the company then moved toward metrics that connect directly to business goals, such as user frustration, rather than optimizing isolated components without knowing whether they matter to users. ### North-star metrics determine which model work deserves attention [02:44](https://www.youtube.com/watch?v=7lNZONiZqqw&t=164s) Goron separates north-star metrics from targeted machine-learning metrics. A transcriber can be evaluated with word error rate, while a turn detector can be evaluated for its own accuracy. The team first asks how the full product is performing, then uses those outcomes to decide which models to improve. He warns that a team could spend its time building the best possible transcriber even when an LLM already corrects many transcription errors. ### Interruptions depend on context and several system components [03:48](https://www.youtube.com/watch?v=7lNZONiZqqw&t=228s) Turn detectors predict whether the bot should speak immediately or wait. They can use text, audio, or multimodal inputs. The right timing depends on the conversation and use case, since a customer-service call has a different pace from a conversation with a friend. Goron also connects interruption behavior to LLM latency and voice activity detection, so improving turn detection alone cannot fully solve the problem. ### Specialized turn detectors start with production data and targeted fine-tuning [05:16](https://www.youtube.com/watch?v=7lNZONiZqqw&t=316s) HappyRobot began with heuristics such as silence, fillers, and punctuation, which provide useful signals about whether a sentence has ended. It then explored multimodal machine-learning models and a production workflow based on observing real calls, adding monitoring, and fine-tuning models for specific use cases. The goal is to understand why interruptions differ across scenarios and adapt the system accordingly. ### LLM annotation scales data work when humans define the standard [06:46](https://www.youtube.com/watch?v=7lNZONiZqqw&t=406s) Training these models requires accurate labels. HappyRobot combines human annotation with LLM-based annotation. The team first builds test sets with human labels, then aligns the LLM annotators' prompts with that human signal. Goron describes this as closing the loop: the company can create accurate datasets across use cases and use them to improve turn detectors. ### Evaluation needs human-aligned judges and model-level monitoring [07:38](https://www.youtube.com/watch?v=7lNZONiZqqw&t=458s) HappyRobot is building a monitoring platform for voice agents. LLMs can judge transcripts or audio for north-star metrics, but those metrics need to be defined carefully and aligned with human annotators. Alongside them, the platform tracks model metrics such as word error rate and turn-detector accuracy. Dashboards show which calls fail and which models contribute to those failures, after which the data can be cleaned and used to improve models. ### Cascading voice systems remain useful because they expose their parts [11:28](https://www.youtube.com/watch?v=7lNZONiZqqw&t=688s) Goron says cascading systems currently have practical advantages over speech-to-speech models. Audio tokens are expensive, and speech-to-speech systems are not yet as good at tool calling as the language models used in cascading systems. Cascading architectures also let teams inspect and monitor transcription, voice activity detection, turn detection, and other components separately instead of treating the system as a black box. ### Teams can start with existing voice frameworks and add guardrails around the LLM [13:12](https://www.youtube.com/watch?v=7lNZONiZqqw&t=792s) For teams testing product demand, Goron recommends building with existing frameworks such as LiveKit and Pipecat rather than waiting to engineer every component from scratch. He says these tools can take teams far, although choosing and tuning models for each language still requires work. For guardrails, HappyRobot focuses mainly on the agent's LLM, since that is where the system's reasoning occurs. TTS hallucinations happen less often, according to Goron. ## Notable quotes - "Our main idea now is to reverse the order: we want to make sure that we understand the north-star metrics and then we can prioritize the features and which models we want to improve based on those ones." (03:28) - "We managed to create really accurate data sets across many use cases and we kind of closed the loop of improving these turn detectors for different use cases." (07:07) - "With this cascading system you have much more control and much more visibility on what's happening in all parts." (12:30) - "I would honestly suggest that they build their own." (13:52) ## Tools & references mentioned - HappyRobot - MLOps Community - OpenAI - LiveKit - Pipecat - DSPy - VAD - voice activity detector ## Who should watch - You are building a voice agent and need to connect call quality metrics with the individual models that affect them. - Your team is deciding whether to use LLMs, humans, or a combination of both for production data annotation. - You are evaluating cascading voice architectures, turn detection, interruption handling, and model monitoring. ## Editor's note Erik Goron describes how HappyRobot uses production calls, LLM annotations, and human labels to create data for improving turn detectors across use cases. ZenML records each pipeline step, its inputs, outputs, and code version, so the data preparation and fine-tuning workflow can be traced back to the run that produced each model or artifact. Written by the MLOps Talks editors (the ZenML team), not by the speaker. ## Related talks - [Voice Agent Use Cases](https://mlopstalks.com/talks/voice-agent-use-cases) (Anurag Beniwal, ElevenLabs, 51:05) - [Building Real-Time, Reliable Voice AI: From Simulation to Production](https://mlopstalks.com/talks/building-real-time-reliable-voice-ai-from-simulation-to-production) (Brooke Hopkins, Coval, 57:13) - [AI Agents for Consumers](https://mlopstalks.com/talks/ai-agents-for-consumers) (Shaun Wei, RealChar, 57:27) - [MLOps at the Crossroads](https://mlopstalks.com/talks/mlops-at-the-crossroads) (Patrick Barker, Kentauros AI & Farhood Etaati, AIMedic, 49:02) - [Who's MLOps for Anyway?](https://mlopstalks.com/talks/whos-mlops-for-anyway) (Jonathan Rioux, EPAM Systems, 1:10:15)