Real-time voice agents need orchestration across latency, consistency, context, and recovery, rather than a simple speech-to-text, LLM, and text-to-speech pipeline.
2
Elyos keeps workflows reliable by combining deterministic backend state machines with model-driven steps, while limiting context to the information needed for the current turn.
3
Voice agents should be judged by groundedness, interruptions, repetition, sentiment, task outcomes, and the failures that occur most often, with a human escalation path available.
Summary
Panos Stravopodis explains how Elyos AI builds voice agents for home-services companies. The agents connect to customer CRM and ERP systems and handle calls, messages, bookings, invoices, payments, engineer dispatch, and follow-up. He describes real-time voice as an infrastructure problem as much as a language problem. Speech-to-text, LLM, and text-to-speech latency accumulates quickly, while noise, interruptions, changing intent, and long calls make simple workflows unreliable. Elyos addresses this with regional deployments, provider routing, warm workers, close placement of tools and orchestration, and monitoring of generated speech. Stravopodis recommends concise workflows, just-in-time context, backend state machines, explicit error handling, tool-result cleanup, post-call reconciliation, and human escalation. He is also direct about trade-offs: orchestration frameworks can add latency, end-to-end models are improving but still have limitations, and non-English voice quality varies. The talk gives practical guidance for teams deploying voice agents where mistakes affect real customers and payments.
Voice agents have to handle real service workflows, not isolated conversations
Elyos connects its agents to customers' CRM and ERP systems so they can manage the full home-services interaction. A customer with a broken boiler might need triage, an emergency appointment, an invoice, payment collection, engineer notification, and resolution tracking. The agent handles calls, emails, and messages around the clock. These workflows are specialized for plumbers, electricians, HVAC installers, and similar businesses. The example shows why a voice agent needs access to business systems and operational steps, rather than only answering questions.
Latency comes from the whole pipeline and quickly affects conversation quality
Stravopodis breaks the voice stack into speech-to-text, an LLM, and text-to-speech, then explains why their delays add up. Speech-to-text may take 100 to 300 milliseconds. Text-to-speech can range from about 200 milliseconds to 500 or 600 milliseconds depending on the voice, language, and hosting location. He gives a best-case P90 range of roughly 500 to 1,200 milliseconds for the pipeline. Noise, unclear audio, and changing user intent make the problem harder. He says teams must measure the whole interaction instead of assuming that connecting three components is enough.
Orchestration reduces latency and keeps the real-time system available
Stravopodis describes latency, consistency, context, and recovery as the four pillars of the system. Workers should be warm when a call starts, and the infrastructure should drain long-running calls safely during deployments. Regional clusters keep pipelines near customers and telephone providers. Tools should stay close to the orchestration layer to reduce network delay. Elyos also routes between provider endpoints because latency can change by location and time of day. Text-to-speech needs its own monitoring because an inconsistent voice can make a conversation sound unnatural even when LLM latency looks good.
Short, controlled workflows make model behavior more predictable
Stravopodis advises teams to define expected outcomes, involve people when a flow goes wrong, and avoid trying to make one runtime perform too many jobs. He recommends avoiding retrieval when it is unnecessary because an incorrect retrieved passage can introduce errors into the conversation. In a known workflow, the required information can often be injected directly into the prompt. Elyos also compares its workflows with human agents rather than assuming humans automatically perform better. The goal is to reduce unnecessary variability in journeys that may involve payments, invoices, or other sensitive actions.
Just-in-time context keeps the prompt small and current
The suggested context pattern starts with a minimal prompt and adds information based on intent and classification. Stravopodis warns that subagents can lose information because the main agent may not know what happened inside them, so the backend needs control of the overall context. Tool calls should be summarized after use and removed when they are no longer needed. Old context can confuse a later call to the same tool, especially when the parameters have changed. In an emergency callout workflow, the agent may only need the result that a case is an emergency after the classification is complete, rather than the full reasoning and history that produced it.
Backend state machines give models room to act without giving them control of sensitive state
Elyos treats workflows as state machines and uses clear success and error conditions to guide recovery. A flow can combine deterministic steps with model-driven steps. For a boiler-service booking, identifying the customer and creating the job can be deterministic, while an LLM can assess service coverage, response time, and cost. Sensitive actions such as invoicing and payments should not be updated directly by the agent. The backend controls the transitions and sends useful error feedback back to the model so it can try a different action.
Recovery needs uncertainty handling and a human path
The system should reconcile the result after a runtime finishes and check whether the intended outcome actually happened. Stravopodis points out that speech-to-text is still weak at details such as UK email addresses and postcodes. Instead of repeatedly annoying the caller, the workflow needs an uncertainty path and a human escalation option. A human can receive a transferred call, provide instructions inside the workflow, or approve part of an action. Elyos also runs a fast model alongside the conversation as a judge, using it to detect when the interaction is drifting and pull the main model back toward the intended path.
Voice-agent quality needs outcome and human-feedback metrics
Time to first token remains useful, but Stravopodis also tracks groundedness, hallucinations, adherence to scripts and workflows, interruptions, repeated words, and whether the requested task was completed. Caller sentiment and the caller's view of how the request was handled provide additional feedback. He recommends focusing first on the most common failure cases because fixing those can improve the system quickly. In the discussion, he says that about 15% of calls on average require some human touchpoint, although the rate varies by customer and does not always mean a transfer.
