Rivia is designed to make routine phone calls, such as customer service and appointment scheduling, so people do not have to spend hours handling them.
2
RealChar combines generative AI with traditional machine learning, running components in parallel through an event bus so different systems can process signals and return actions.
3
Making an AI agent reliable requires controlled testing, fallbacks, monitoring, benchmarks, evaluations, and human takeover because generative models can be slow, inconsistent, or updated without warning.
Summary
Shaun Wei describes Rivia, RealChar's consumer assistant for handling routine phone calls. The product is intended to call customer service lines, press keypad buttons, receive text messages, schedule appointments, and report the result to the user. Shaun draws on his work on Google Duplex and self-driving cars. Duplex showed him how difficult natural phone conversations are, while autonomous driving taught him about simulation, fast processing, multimodal inputs, and recovery when a model fails. Rivia processes audio, text, button presses, and other signals in parallel. Traditional machine learning models identify intents and triggers, while generative models use the accumulated context to produce responses. The system has model fallbacks, real-time tracing, and a virtual testing environment. Shaun is candid that the product is still in beta and that production quality is much harder than a polished demo. He expects more interaction data to support dedicated models for specific use cases over time.
Rivia takes routine phone calls away from the user
Shaun Wei says Rivia is being prepared for the App Store and is still in a small beta test. The intended assistant can make phone calls for users and handle tasks that people find boring or blocking. Examples include dealing with a utility company, contacting customer support, and arranging a plumber's visit. Demetrios Brinkmann describes a user receiving a text with the appointment window after the assistant finishes the call. Shaun frames the purpose around giving people more time for family, work, and other meaningful activities instead of spending hours with plumbers or utility companies. He wants the service to work across regions, although Demetrios points out that languages and cultural differences make that ambitious.
Google Duplex exposed the difficulty of natural phone conversations
Shaun Wei worked on both sides of Google Duplex, building the system and interviewing people who received its calls. In 2018, the team used natural-language processing and natural-language understanding models for phone conversations, with extensive labeled data needed to cover many conversational cases. The system captured user intents, mapped them to actions, and combined speech-to-text and text-to-speech components with classification models. A hair salon example showed how quickly this approach could fail. The system could handle a basic haircut booking, then break when asked whether the customer wanted a male or female haircut. Shaun says this experience showed him that people wanted an assistant to take over boring phone calls, while also exposing how many small cases a system must handle.
Shaun connects Rivia's design to lessons from self-driving cars. He describes autonomous driving as a highly integrated system that must behave around humans and operate in the real world. One lesson was to simulate situations in a virtual environment, where a system can encounter failures repeatedly without putting people at risk. Another was processing information very quickly while still allowing the system to scale. He also learned to combine multiple modalities. A self-driving car uses camera and lidar data, and may use audio to detect sirens and decide how to pull over safely. Shaun applies similar ideas to phone calls, where audio is only one part of the interaction.
Shaun says a phone call includes audio, keypad presses, and incoming text messages. A customer-service system may send a link or a verification code by text, so the assistant must understand that new input and use it during the call. Rivia's perception models accept different streams and process them as they arrive. Shaun compares this with video and audio streaming into a robotics backend. The system can process batches such as every two frames or a fixed amount of audio, depending on the required speed. He describes a clock-cycle design in which the system processes recent data at regular intervals and generates signals for downstream components.
Traditional machine learning and generative models share the work
Shaun says the long-term goal is an end-to-end model that can accept different data types and map them to actions. Rivia is not there yet because it is still collecting data. Its current design uses traditional NLP and machine-learning components to identify intents, triggers, and signals. Generative AI models then use those signals and the surrounding context to produce a response. Shaun compares this context-building step with retrieval-augmented generation, although the phone system gathers information from the live interaction rather than a vector database. Validation and context expansion help the generative model stay focused on the user's task. The output can be text-to-speech or another direct response, depending on the action.
Parallel processing and fallbacks keep calls responsive
Rivia uses several systems at once rather than waiting for every task to pass through one model. Shaun describes an event bus where components subscribe to events they can handle and publish their responses for downstream systems. A gateway also routes different requests and intents toward the appropriate systems. The design includes a priority for some requests, but it can choose whether to wait, press a button, or use another path. If a preferred model is too slow, a fallback can take over. Shaun compares this with a self-driving car that recovers when an AI model fails instead of stopping in the road. This matters especially for audio, where users notice even a short delay or a cut-off response.
Production reliability is harder than an impressive demo
Shaun calls generative AI systems difficult to productionize because responses can be slow, inconsistent, or different after a provider changes its model. A system may work one day and behave differently the next. He says a demo might work once in a hundred attempts, while making the overall system work reliably demands much more engineering effort. RealChar uses a controlled virtual environment with recorded human interactions to generate synthetic data and validate behavior. The team can inspect what happened at a particular millisecond and ask whether the problem came from a generative model, a traditional machine-learning model, or an external service going down. Shaun also mentions monitoring, benchmarks, and evaluations as part of the work required before exposing an agent to the public.
The beta will collect interaction data for more focused models
Shaun expects real interactions to make the system smarter over time. He wants to launch early so the team can observe how the AI and people interact during phone calls. Over time, he expects dedicated models for different use cases, such as utility-company calls or other forms of customer support. He does not claim that Rivia can handle every situation today. His comparison is a self-driving car that can reliably drive in a straight line for a limited distance but is not ready for every highway detour. The product is offered through a flat subscription model, so users do not need to decide whether each individual phone call is worth the cost.
"You can build a really cool demo, probably work only one out of 100 times, but if you make sure everything is working 90% of the time, that is a very hard engineering problem to solve."Shaun Wei42:54
Who should watch
You are building a phone or voice agent and need to understand why latency, audio streaming, and fallbacks become engineering problems quickly.
Your team has a convincing generative-AI demo but needs a testing setup, monitoring, and a plan for model failures before releasing it to users.
You want to explore consumer automation for customer-service calls and are comfortable evaluating a beta product with limited coverage.
