Runway uses machine learning in a browser-based video editor so users can apply effects without owning powerful hardware.
2
The system races a user's playhead by fetching, decoding, processing, encoding, and streaming video before the edited media fully exists.
3
Brannon's team keeps compute workloads stateless and deploys frequently so it can use preemptible capacity and shorten the feedback loop.
Summary
Brannon Dorsey describes Runway as a web-based video editor with machine learning tools for tasks such as rotoscoping, inpainting, and audio cleanup. The company began as a model zoo that let creatives run more than 100 image-based PyTorch and TensorFlow models, then focused on rotoscoping after seeing that users valued a specific professional workflow more than access to many models. Runway's backend must process uploaded video quickly enough for a user to press play and see an effect while the output is still being generated. That means moving media from object storage through decoding, frame processing, model inference, encoding, and browser streaming. Brannon explains why the company chose the web, including easier access, collaboration, shared cloud media, and support for weak devices. He also discusses Kubernetes, stateless workers, checkpoints, spot instances, and frequent production releases. He is candid that cloud cost and infrastructure remain ongoing trade-offs.
Runway uses machine learning to make browser-based video editing accessible
Brannon describes Runway as a video editor that runs on the web and uses machine learning behind the scenes. Its specialized tools automate editing tasks or add capabilities through remote server infrastructure. The product targets professional editors and people making short-form social media videos who may know iMovie but have never used Premiere. Green Screen can cut a person out of video and propagate that edit across the whole clip. Brannon says a task that could take five hours can take five minutes with the tool.
The company moved from a model zoo to one focused workflow
Runway began as a desktop model zoo for creatives. The team containerized PyTorch and TensorFlow models behind a standard interface, supported more than 100 models, and let users run them on Runway's GPUs or on their own hardware. Brannon felt the product risked becoming a collection of demonstrations rather than a professional tool. The company changed direction by choosing one use case, rotoscoping, and putting its resources into making that experience work well. The response from users changed how the company built its product.
Rotoscoping was the first focused bet because it removed a manual bottleneck
Brannon says the team chose rotoscoping because it believed Runway could do it better than existing tools. The intended workflow was to edit a single frame and propagate the change through a video, rather than manually repeating the process. Rotoscoping is laborious, so a tool that worked quickly and produced a good-looking result could change how people edited. The team later expanded from that focused tool into a full linear video editor in the browser.
Runway races the playhead through a live video processing pipeline
Runway's main engineering problem is on-the-fly inference over video. A user uploads a clip to object storage, applies an effect, and expects to press the spacebar and see playback before the processed media fully exists. The backend fetches compressed media, decodes it into NumPy frames, runs models over batches, re-encodes the result, and streams it back to the browser. Users then make further edits, creating a human-in-the-loop feedback cycle. The system must handle heavy computation and spiky traffic while keeping the experience responsive and reliable.
User response time defines the system's requirements
Brannon says video is difficult to test because quality is often qualitative rather than a simple numerical output. Runway therefore starts from the editing experience: if a user makes a change, the system must apply it within a small attention window. The team watches its compute cluster and queuing systems, but its main metric is user response time. It sets targets such as keeping p95 latency below a chosen threshold. Green Screen initially ran at about 4 fps, yet users still valued it enough to keep using it, which showed the team that the workflow was worth optimizing.
Video models operate on a time-stacked collection of frames
When Vishnu asks what a model sees, Brannon explains that video adds a time dimension to the usual image representation. The input is generally much larger than 256 by 256, often 1920 by 1080 or 4K, with three color channels and a stack of frames over time. Models usually receive that pile of frames, perform an operation, and produce another pile of frames. A decoder sits before the model and an encoder follows it. Brannon even describes a pass-through decoder-and-encoder pipeline as the simplest model because it streams video without changing it.
The browser lowers access and makes collaboration easier
Brannon gives several reasons for building Runway in the browser. A user can follow a link and start without downloading or learning a desktop application. The browser also fits collaboration patterns familiar from Google Docs. Runway uploads media to cloud storage so it can be reached from different devices and shared with other people, avoiding missing-media problems caused by local files. The team found that only about 2 percent of users had hardware that could compete with its cloud GPUs. It therefore aims to work well even on a five-year-old Chromebook.
Stateless workers let Runway tolerate interruption and reduce compute cost
Runway keeps compute units, such as pods, stateless where possible. Workers take jobs from stateful systems such as managed databases or object storage, report progress, and save checkpoints. A worker can disappear and another can continue the work. This improves availability and lets the company use spot instances, whose compute can be taken away with a short warning, at a lower cost. Brannon says cloud cost remains a real concern, and the team has learned to manage the trade-off between an ideal backend and the user experience it wants to provide.
Frequent production releases make MLOps an extension of DevOps
Brannon argues that good MLOps often starts with good DevOps practices. Runway tries to get code changes from authoring to production quickly, which shortens the loop between building a change, observing it in production, and seeing how users respond. Releasing around a dozen times a day makes releases less precious and lets engineers revert or follow up with another change. Branches map to meaningful environments, pull requests are tested in staging, and merging to master releases to production. The Git history then provides a way to reason about what is deployed and when bugs may have appeared.
