LiveKit began as an open-source project that made real-time audio and video infrastructure easier for developers to use during the pandemic.
2
LiveKit became an AI company after OpenAI used Russ d'Sa's voice demo to build ChatGPT's first voice mode.
3
Russ d'Sa sees LiveKit as infrastructure connecting people to AI through voice, video, and eventually physical movement.
Summary
Russ d'Sa explains how LiveKit started in 2020 as an open-source answer to the difficulty of building real-time audio and video applications. Companies including Spotify, Adobe, eBay, and Oracle adopted it, which led LiveKit to build a paid cloud network. After ChatGPT launched, d'Sa built an improvised voice interface and posted it online. Five months later, OpenAI contacted LiveKit because it had found the demo and wanted to build a real voice interface for ChatGPT. Shipping the first version of voice mode changed how d'Sa understood the company. He now describes LiveKit as infrastructure around AI models, handling communication, voice agents, deployment, scaling, and observability. He also explains why voice is difficult: timing, interruptions, intonation, silence, tool calls, and model latency all affect whether an interaction feels natural. The right latency depends on the use case, and systems need to signal when they are thinking or busy.
LiveKit started by solving a pandemic-era developer problem
LiveKit began around the start of the pandemic, when people relied on cameras and microphones to communicate online. Russ d'Sa found that the internet's usual text-oriented infrastructure was poorly suited to real-time audio and video. WebRTC existed, but most developers still had to build the surrounding infrastructure themselves. While working on a company version of Clubhouse, d'Sa built that infrastructure and realized other developers could use it. He and his co-founder David released it as an open-source project for real-time audio and video streaming.
Early adoption created demand for a commercial cloud product
Spotify, Adobe, eBay, Oracle, and other large companies started using LiveKit quickly. d'Sa says this looked like product-market fit, although LiveKit only had an open-source project and no commercial product. Those companies told the founders they did not want to deploy, scale, and operate the system themselves. They wanted LiveKit to run the network for them. That request led the company to raise money and build LiveKit Cloud, a global network for streaming audio and video, launched at the end of 2022.
A failed voice demo later brought OpenAI to LiveKit
After ChatGPT launched, d'Sa wondered whether LiveKit could be combined with the website to let people talk to ChatGPT. He connected speech recognition, a headless browser, and the ChatGPT interface in a rough demo, then posted it on Twitter. It did not go viral. Five months later, a large cloud provider asked whether it could buy or license LiveKit, or compete with it through open source. Soon after, OpenAI contacted d'Sa because it had found his blog post and demo. OpenAI had already signed up with a personal Gmail account and wanted to build the voice interface for real.
LiveKit spent about a month working with OpenAI and shipped the first version of voice mode in September, before later versions such as Advanced Voice and GPT-4o. That project made d'Sa realize LiveKit was an AI company. His reasoning is that if AI labs build a synthetic human or human-like computer, people will interact with it through cameras, microphones, and speakers. LiveKit already connected humans through those same devices, so it could connect a person to a machine instead.
LiveKit wants to provide the infrastructure around AI models
d'Sa describes AI labs such as OpenAI, Anthropic, and Gemini as building the brain, while LiveKit can carry signals to and from it. He argues that large language models are probabilistic computers and need ordinary, deterministic software around them before developers can turn them into useful products. LiveKit's focus has shifted toward infrastructure for voice, video, and physical AI. The aim is to give applications the ability to see, hear, speak, and, in robotics, move.
Voice agents need tools for testing, deployment, and observation
LiveKit has a transport network and an agents framework for building voice agents, but d'Sa says the complete platform does not exist yet. Developers still need ways to test and evaluate agents, deploy and load-balance them, host their stateful sessions, and observe user experiences. Voice agents differ from ordinary web applications because they remain active for the length of a session and continuously process and generate voice. LiveKit is working on hosting and exploring evaluation partnerships or features that fit its ecosystem. d'Sa compares the agents framework to Next.js for voice agents and asks what the equivalent of Datadog or New Relic would be for voice AI.
Voice carries more information than text and is harder to reproduce
Voice includes pronunciation, intonation, cadence, and other sound qualities that affect how people interpret meaning. That makes voice a rich and natural medium, but it also makes synthetic conversation difficult to reproduce in code. LiveKit's framework handles interruptions, turn-taking, and low-latency streaming, while the rest of the system still has to manage transcription quality, prompts, tool calls, response timing, and conversational dynamics. If those pieces fail, users can reject the experience as unnatural. The tolerance for problems depends on the application.
Silence makes waiting feel broken in a voice interaction
Russ d'Sa contrasts patient intake with language learning. A patient may accept a one- or two-second delay because a human receptionist might be checking a calendar or writing something down. Language learners expect faster responses because the interaction is meant to feel like a conversation with a native speaker. d'Sa says silence is especially damaging. Human operators provide signals such as paper sounds, keyboard noise, or a short explanation that they are checking something. A voice agent should provide similar feedback, especially when a reasoning model needs time to think.
"If OpenAI and Anthropic and Gemini and the labs are going to build the brain, LiveKit has an opportunity to build the nervous system to carry signals to and from that brain wherever they originate."Russ d'Sa12:16
Who should watch
You are building a voice agent and need to understand the infrastructure problems that appear after the model works.
You want a practical account of how an open-source real-time communications project moved into AI infrastructure.
Your team is making voice interactions feel natural and needs guidance on latency, silence, turn-taking, and feedback.
