# Real-time Model Inference in a Video Streaming Environment

Brannon Dorsey, Runway | MLOps Coffee Sessions | Episode 98 | 58:02
Hosted by Vishnu Rachakonda

Source: https://www.youtube.com/watch?v=TNO6rYwP3yg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/real-time-model-inference-in-a-video-streaming-environment
Published: 2022-05-12
Tags: cost, deployment, latency, model-serving

## TL;DR
- Runway uses machine learning in a browser-based video editor so users can apply effects without owning powerful hardware.
- The system races a user's playhead by fetching, decoding, processing, encoding, and streaming video before the edited media fully exists.
- Brannon's team keeps compute workloads stateless and deploys frequently so it can use preemptible capacity and shorten the feedback loop.

## Summary
Brannon Dorsey describes Runway as a web-based video editor with machine learning tools for tasks such as rotoscoping, inpainting, and audio cleanup. The company began as a model zoo that let creatives run more than 100 image-based PyTorch and TensorFlow models, then focused on rotoscoping after seeing that users valued a specific professional workflow more than access to many models. Runway's backend must process uploaded video quickly enough for a user to press play and see an effect while the output is still being generated. That means moving media from object storage through decoding, frame processing, model inference, encoding, and browser streaming. Brannon explains why the company chose the web, including easier access, collaboration, shared cloud media, and support for weak devices. He also discusses Kubernetes, stateless workers, checkpoints, spot instances, and frequent production releases. He is candid that cloud cost and infrastructure remain ongoing trade-offs.

## Key ideas
### Runway uses machine learning to make browser-based video editing accessible
[05:42](https://www.youtube.com/watch?v=TNO6rYwP3yg&t=342s)
Brannon describes Runway as a video editor that runs on the web and uses machine learning behind the scenes. Its specialized tools automate editing tasks or add capabilities through remote server infrastructure. The product targets professional editors and people making short-form social media videos who may know iMovie but have never used Premiere. Green Screen can cut a person out of video and propagate that edit across the whole clip. Brannon says a task that could take five hours can take five minutes with the tool.

### The company moved from a model zoo to one focused workflow
[11:07](https://www.youtube.com/watch?v=TNO6rYwP3yg&t=667s)
Runway began as a desktop model zoo for creatives. The team containerized PyTorch and TensorFlow models behind a standard interface, supported more than 100 models, and let users run them on Runway's GPUs or on their own hardware. Brannon felt the product risked becoming a collection of demonstrations rather than a professional tool. The company changed direction by choosing one use case, rotoscoping, and putting its resources into making that experience work well. The response from users changed how the company built its product.

### Rotoscoping was the first focused bet because it removed a manual bottleneck
[14:37](https://www.youtube.com/watch?v=TNO6rYwP3yg&t=877s)
Brannon says the team chose rotoscoping because it believed Runway could do it better than existing tools. The intended workflow was to edit a single frame and propagate the change through a video, rather than manually repeating the process. Rotoscoping is laborious, so a tool that worked quickly and produced a good-looking result could change how people edited. The team later expanded from that focused tool into a full linear video editor in the browser.

### Runway races the playhead through a live video processing pipeline
[16:46](https://www.youtube.com/watch?v=TNO6rYwP3yg&t=1006s)
Runway's main engineering problem is on-the-fly inference over video. A user uploads a clip to object storage, applies an effect, and expects to press the spacebar and see playback before the processed media fully exists. The backend fetches compressed media, decodes it into NumPy frames, runs models over batches, re-encodes the result, and streams it back to the browser. Users then make further edits, creating a human-in-the-loop feedback cycle. The system must handle heavy computation and spiky traffic while keeping the experience responsive and reliable.

### User response time defines the system's requirements
[20:17](https://www.youtube.com/watch?v=TNO6rYwP3yg&t=1217s)
Brannon says video is difficult to test because quality is often qualitative rather than a simple numerical output. Runway therefore starts from the editing experience: if a user makes a change, the system must apply it within a small attention window. The team watches its compute cluster and queuing systems, but its main metric is user response time. It sets targets such as keeping p95 latency below a chosen threshold. Green Screen initially ran at about 4 fps, yet users still valued it enough to keep using it, which showed the team that the workflow was worth optimizing.

### Video models operate on a time-stacked collection of frames
[24:06](https://www.youtube.com/watch?v=TNO6rYwP3yg&t=1446s)
When Vishnu asks what a model sees, Brannon explains that video adds a time dimension to the usual image representation. The input is generally much larger than 256 by 256, often 1920 by 1080 or 4K, with three color channels and a stack of frames over time. Models usually receive that pile of frames, perform an operation, and produce another pile of frames. A decoder sits before the model and an encoder follows it. Brannon even describes a pass-through decoder-and-encoder pipeline as the simplest model because it streams video without changing it.

### The browser lowers access and makes collaboration easier
[25:40](https://www.youtube.com/watch?v=TNO6rYwP3yg&t=1540s)
Brannon gives several reasons for building Runway in the browser. A user can follow a link and start without downloading or learning a desktop application. The browser also fits collaboration patterns familiar from Google Docs. Runway uploads media to cloud storage so it can be reached from different devices and shared with other people, avoiding missing-media problems caused by local files. The team found that only about 2 percent of users had hardware that could compete with its cloud GPUs. It therefore aims to work well even on a five-year-old Chromebook.

### Stateless workers let Runway tolerate interruption and reduce compute cost
[30:11](https://www.youtube.com/watch?v=TNO6rYwP3yg&t=1811s)
Runway keeps compute units, such as pods, stateless where possible. Workers take jobs from stateful systems such as managed databases or object storage, report progress, and save checkpoints. A worker can disappear and another can continue the work. This improves availability and lets the company use spot instances, whose compute can be taken away with a short warning, at a lower cost. Brannon says cloud cost remains a real concern, and the team has learned to manage the trade-off between an ideal backend and the user experience it wants to provide.

### Frequent production releases make MLOps an extension of DevOps
[38:45](https://www.youtube.com/watch?v=TNO6rYwP3yg&t=2325s)
Brannon argues that good MLOps often starts with good DevOps practices. Runway tries to get code changes from authoring to production quickly, which shortens the loop between building a change, observing it in production, and seeing how users respond. Releasing around a dozen times a day makes releases less precious and lets engineers revert or follow up with another change. Branches map to meaningful environments, pull requests are tested in staging, and merging to master releases to production. The Git history then provides a way to reason about what is deployed and when bugs may have appeared.

## Notable quotes
- Brannon Dorsey: "We really operate on this human in the loop kind of approach to ML." (19:08)
- Brannon Dorsey: "Ultimately like the golden metric is response time to users and user experience in when using the app." (21:59)
- Brannon Dorsey: "We want our product to run great on a Chromebook from five years ago." (29:08)
- Brannon Dorsey: "Good MLOps in a lot of ways starts with good DevOps and it's kind of an extension of DevOps." (38:45)
- Brannon Dorsey: "Don't come to the product because it's AI or ML, come to the product because it's useful in your life." (53:02)

## Tools & references mentioned
- Runway
- PyTorch
- TensorFlow
- Premiere
- DaVinci
- iMovie
- Netflix
- Google Docs
- Kubernetes
- AWS
- RDS
- Spot by NetApp
- AWS Fargate
- Docker
- Linode
- Dungeons & Dragons
- Generative adversarial networks

## Who should watch
- You are building an interactive ML product where users need to see results before a long-running job has finished.
- Your video or other media workloads create large, spiky inference jobs and you are weighing stateless workers, checkpoints, and spot instances.
- You want practical examples of how release practices, infrastructure choices, and user experience targets shape an MLOps system.

## Editor's note

Brannon Dorsey says Runway keeps compute workers stateless so interrupted video jobs can continue elsewhere and the team can use spot instances. ZenML runs workflows as Python steps on a configurable stack, so the same pipeline code can run with different infrastructure choices without being rewritten. Each run also records its steps, inputs, outputs, and code version.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Scaling Real-time Machine Learning at Chime](https://mlopstalks.com/talks/scaling-real-time-machine-learning-at-chime) (Peeyush Agarwal, Chime, 24:22)
- ["Real-Time" ML: Features and Inference](https://mlopstalks.com/talks/real-time-ml-features-and-inference) (Sasha Ovsankin & Rupesh Gupta, LinkedIn, 51:55)
- [Real-time Machine Learning](https://mlopstalks.com/talks/real-time-machine-learning) (Chip Huyen, Claypot AI, 58:24)
- [Exploring the Latency/Throughput & Cost Space for LLM Inference](https://mlopstalks.com/talks/exploring-the-latency-throughput-cost-space-for-llm-inference) (Timothée Lacroix, Mistral, 30:25)
- [The 7 Lines of Code You Need to Run Faster Real-time Inference](https://mlopstalks.com/talks/the-7-lines-of-code-you-need-to-run-faster-real-time-inference) (Adrian Boguszewski, Intel, 49:21)