# An AI Company By Accident

Russ d'Sa, LiveKit | MLOps Community | 32:35
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=FeXvbC3SZYY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/an-ai-company-by-accident
Published: 2025-09-21
Tags: latency, observability, open-source, voice

## TL;DR
- LiveKit began as an open-source project that made real-time audio and video infrastructure easier for developers to use during the pandemic.
- LiveKit became an AI company after OpenAI used Russ d'Sa's voice demo to build ChatGPT's first voice mode.
- Russ d'Sa sees LiveKit as infrastructure connecting people to AI through voice, video, and eventually physical movement.

## Summary
Russ d'Sa explains how LiveKit started in 2020 as an open-source answer to the difficulty of building real-time audio and video applications. Companies including Spotify, Adobe, eBay, and Oracle adopted it, which led LiveKit to build a paid cloud network. After ChatGPT launched, d'Sa built an improvised voice interface and posted it online. Five months later, OpenAI contacted LiveKit because it had found the demo and wanted to build a real voice interface for ChatGPT. Shipping the first version of voice mode changed how d'Sa understood the company. He now describes LiveKit as infrastructure around AI models, handling communication, voice agents, deployment, scaling, and observability. He also explains why voice is difficult: timing, interruptions, intonation, silence, tool calls, and model latency all affect whether an interaction feels natural. The right latency depends on the use case, and systems need to signal when they are thinking or busy.

## Key ideas
### LiveKit started by solving a pandemic-era developer problem
[01:37](https://www.youtube.com/watch?v=FeXvbC3SZYY&t=97s)
LiveKit began around the start of the pandemic, when people relied on cameras and microphones to communicate online. Russ d'Sa found that the internet's usual text-oriented infrastructure was poorly suited to real-time audio and video. WebRTC existed, but most developers still had to build the surrounding infrastructure themselves. While working on a company version of Clubhouse, d'Sa built that infrastructure and realized other developers could use it. He and his co-founder David released it as an open-source project for real-time audio and video streaming.

### Early adoption created demand for a commercial cloud product
[05:21](https://www.youtube.com/watch?v=FeXvbC3SZYY&t=321s)
Spotify, Adobe, eBay, Oracle, and other large companies started using LiveKit quickly. d'Sa says this looked like product-market fit, although LiveKit only had an open-source project and no commercial product. Those companies told the founders they did not want to deploy, scale, and operate the system themselves. They wanted LiveKit to run the network for them. That request led the company to raise money and build LiveKit Cloud, a global network for streaming audio and video, launched at the end of 2022.

### A failed voice demo later brought OpenAI to LiveKit
[06:46](https://www.youtube.com/watch?v=FeXvbC3SZYY&t=406s)
After ChatGPT launched, d'Sa wondered whether LiveKit could be combined with the website to let people talk to ChatGPT. He connected speech recognition, a headless browser, and the ChatGPT interface in a rough demo, then posted it on Twitter. It did not go viral. Five months later, a large cloud provider asked whether it could buy or license LiveKit, or compete with it through open source. Soon after, OpenAI contacted d'Sa because it had found his blog post and demo. OpenAI had already signed up with a personal Gmail account and wanted to build the voice interface for real.

### Shipping ChatGPT voice mode changed LiveKit's identity
[10:47](https://www.youtube.com/watch?v=FeXvbC3SZYY&t=647s)
LiveKit spent about a month working with OpenAI and shipped the first version of voice mode in September, before later versions such as Advanced Voice and GPT-4o. That project made d'Sa realize LiveKit was an AI company. His reasoning is that if AI labs build a synthetic human or human-like computer, people will interact with it through cameras, microphones, and speakers. LiveKit already connected humans through those same devices, so it could connect a person to a machine instead.

### LiveKit wants to provide the infrastructure around AI models
[12:16](https://www.youtube.com/watch?v=FeXvbC3SZYY&t=736s)
d'Sa describes AI labs such as OpenAI, Anthropic, and Gemini as building the brain, while LiveKit can carry signals to and from it. He argues that large language models are probabilistic computers and need ordinary, deterministic software around them before developers can turn them into useful products. LiveKit's focus has shifted toward infrastructure for voice, video, and physical AI. The aim is to give applications the ability to see, hear, speak, and, in robotics, move.

### Voice agents need tools for testing, deployment, and observation
[20:08](https://www.youtube.com/watch?v=FeXvbC3SZYY&t=1208s)
LiveKit has a transport network and an agents framework for building voice agents, but d'Sa says the complete platform does not exist yet. Developers still need ways to test and evaluate agents, deploy and load-balance them, host their stateful sessions, and observe user experiences. Voice agents differ from ordinary web applications because they remain active for the length of a session and continuously process and generate voice. LiveKit is working on hosting and exploring evaluation partnerships or features that fit its ecosystem. d'Sa compares the agents framework to Next.js for voice agents and asks what the equivalent of Datadog or New Relic would be for voice AI.

### Voice carries more information than text and is harder to reproduce
[23:30](https://www.youtube.com/watch?v=FeXvbC3SZYY&t=1410s)
Voice includes pronunciation, intonation, cadence, and other sound qualities that affect how people interpret meaning. That makes voice a rich and natural medium, but it also makes synthetic conversation difficult to reproduce in code. LiveKit's framework handles interruptions, turn-taking, and low-latency streaming, while the rest of the system still has to manage transcription quality, prompts, tool calls, response timing, and conversational dynamics. If those pieces fail, users can reject the experience as unnatural. The tolerance for problems depends on the application.

### Silence makes waiting feel broken in a voice interaction
[26:11](https://www.youtube.com/watch?v=FeXvbC3SZYY&t=1571s)
Russ d'Sa contrasts patient intake with language learning. A patient may accept a one- or two-second delay because a human receptionist might be checking a calendar or writing something down. Language learners expect faster responses because the interaction is meant to feel like a conversation with a native speaker. d'Sa says silence is especially damaging. Human operators provide signals such as paper sounds, keyboard noise, or a short explanation that they are checking something. A voice agent should provide similar feedback, especially when a reasoning model needs time to think.

## Notable quotes
- Russ d'Sa: "And in that moment was when I realized that this is an AI company." (11:09)
- Russ d'Sa: "If OpenAI and Anthropic and Gemini and the labs are going to build the brain, LiveKit has an opportunity to build the nervous system to carry signals to and from that brain wherever they originate." (12:16)
- Russ d'Sa: "You can't engineer that, right? And so it's kind of surreal honestly that it ended up happening." (14:44)
- Russ d'Sa: "If you just hear silence, you go crazy." (29:45)
- Russ d'Sa: "There has to be that feedback that you get, to kind of prime your mind as to what's happening." (32:01)

## Tools & references mentioned
- LiveKit
- WebRTC
- Google Meet
- Discord
- Zoom
- Spotify
- Adobe
- eBay
- Oracle
- Clubhouse
- Medium
- ChatGPT
- OpenAI
- Anthropic
- Gemini
- GPT-3.5
- GPT-4o
- Justin Khan
- Justin TV
- Twitch
- Assort Health
- Speak
- VOCA
- Datadog
- New Relic
- Next.js
- MCP
- Bitcoin

## Who should watch
- You are building a voice agent and need to understand the infrastructure problems that appear after the model works.
- You want a practical account of how an open-source real-time communications project moved into AI infrastructure.
- Your team is making voice interactions feel natural and needs guidance on latency, silence, turn-taking, and feedback.

## Related talks

- [AI Agents for Consumers](https://mlopstalks.com/talks/ai-agents-for-consumers) (Shaun Wei, RealChar, 57:27)
- [Voice and Language Tech](https://mlopstalks.com/talks/voice-and-language-tech) (Catherin Breslin, Kingfisher Labs, 55:16)
- [The Challenge with AI Voice Agents](https://mlopstalks.com/talks/the-challenge-with-ai-voice-agents) (Paul van der Boor & Floris Fok, Prosus Group, 47:38)
- [The Coming Revolution of AI Agents](https://mlopstalks.com/talks/the-coming-revolution-of-ai-agents) (Jazmia Henry, Iso AI & Rogerio Bonatti, Microsoft & Julia Kroll, Deepgram & Joshua Alphonse, PremAI, 28:59)
- [Hugging Face Cofounder on AI Agents, LLMs and Open Source](https://mlopstalks.com/talks/hugging-face-cofounder-on-ai-agents-llms-and-open-source) (Thomas Wolf, Hugging Face & Euro Beinat, Prosus Group, 27:31)