Multimodal training data needs a feature-store-like system that can search, store, and stream videos, images, embeddings, text, and ordinary metadata.
2
Researchers and production engineers work better when they share one codebase for training and inference, with tests and reusable libraries instead of handing over serialized scripts.
3
Large-scale training needs to pull only the relevant batches and columns to each machine, rather than copying huge multimodal datasets to every training worker.
Summary
Ethan Rosenthal describes the infrastructure problems he sees while moving from tabular machine learning and language models to generative video at Runway. His main proposal is a multimodal feature store for training data. It would hold raw media alongside metadata and precomputed features such as embeddings and depth estimates, support semantic search, and stream selected batches to distributed training jobs. Ethan also discusses the working relationship between researchers and engineers. He argues for a shared codebase that supports both training and inference, so teams can reuse libraries, write tests, and avoid throwing a script over a wall for someone else to productionize. He prefers cloud training interfaces that stay close to ordinary Docker-based development and criticizes YAML-heavy systems that force researchers and engineers to build custom configuration layers. Runway has a machine learning acceleration team between research and software engineering, focused on faster experiments, model training, inference, and GPU use.
Runway's video tools combine generation with ordinary creative controls
Ethan explains that Runway generates video from text, animates an image, and offers other video tools for creative work. He is especially interested in tools that do not require constant prompting. Runway's green screen tool lets a user paint a mask over the area to remove, then applies that choice across the whole video. Ethan contrasts this with manually drawing a lasso around a person in every frame. He says the company was founded by creatives, which shaped its focus on user experience and practical tools for content creators. For him, using the computer's full interface is more interesting than writing prompts all day.
A multimodal feature store would organize media and derived training features
Ethan uses multimodal to mean working with images, video, audio, and text as model inputs or outputs. He says these were historically handled separately, while current systems combine several modes. His feature-store idea targets the training side of generative AI. A video dataset might include the video itself, its width and height, its resolution, and other information used during training. The challenge is that media is large and slow to download or decode, while columns can have very different sizes and types. Ethan calls storing, querying, and filtering this data for large training jobs an unsolved problem.
The training problem differs from the low-latency problem in traditional feature stores
Demetrios points out that traditional feature stores are often associated with low-latency online inference, such as fraud detection. Ethan agrees that fast inference is a major difficulty for tabular features, while multimodal generative systems often have fewer inference inputs, such as a prompt and perhaps an image. The harder part shifts to training, where the dataset can contain large videos, images, and embeddings. Traditional feature stores still need training data to match inference data and avoid skew. For multimodal systems, Ethan is primarily describing a training store that can hold and serve large, varied inputs.
Precomputed embeddings and model outputs give researchers reusable training choices
Ethan says multimodal data needs more than storage. Researchers need to search and curate it. Finding images with a semantic property, such as cats, may require image embeddings and nearest-neighbor search rather than a simple SQL query. The store could also contain precomputed features from other models. In Runway's Gen-1 example, text can become a CLIP embedding, while a depth-estimation model can estimate the distance of each pixel from the camera. These calculations can happen during training or beforehand. Storing several versions lets researchers choose different features for experiments without recomputing everything inside every training job.
Distributed training should stream only the needed data to each worker
Ethan describes training jobs that use many machines, GPUs, and data-loader workers. With a small text dataset, a team might copy the whole dataset to every machine. That approach fails for videos, images, and embeddings because the data becomes too large. A separate system could first create a complete training dataset, but that can also waste storage and time. Ethan's preferred design lets the feature store query the rows and columns needed by a job, divide the data across machines, and stream batches as stochastic gradient descent requests them. Workers receive relevant data instead of downloading the whole collection.
A shared training and inference codebase avoids throwing scripts over the wall
Ethan criticizes workflows where researchers serialize a model and hand over a Python script or artifact to a production team. That approach makes researchers rebuild models from scratch and prevents reuse of libraries and shared code. At Square, his team used one codebase for training and inference. They could keep shared code dry, write tests, and use CI/CD. The trade-off is that changes in the training pipeline must avoid breaking other jobs. Ethan accepts that constraint because a shared codebase gives both sides a common interface and makes production work part of normal software development.
Researchers need an existing example and a low-friction path to cloud GPUs
Ethan says researchers often struggle when asked to start with an empty repository, add tests and CI/CD, build a Docker image, and define cloud training from scratch. An existing codebase with examples makes refactoring and adding tests easier. He also prefers tools that keep researchers inside a Docker-based workflow. The ideal experience is writing code locally with the editor of choice, then running the same code in the cloud across many GPUs with caching that avoids repeated uploads and downloads. He mentions Modal and Bow as companies exploring this kind of interface.
Runway's acceleration team sits between research and software engineering
Ethan says Runway has research, backend engineering, frontend engineering, and a machine learning acceleration team between research and backend engineering. The acceleration team works on both training and inference. Faster training lets researchers run experiments sooner, while faster inference allows more models to fit on expensive GPUs and improves capacity use. Ethan says the team does not work from fixed quarterly KPIs because startup priorities change quickly. He describes a mixture of tools and systems rather than an entirely homegrown stack. The team's practical goal is to reduce the friction between research code and large-scale execution.
"In an ideal world you can, your feature store can solve for searching and querying and storing all of your data, but it also solves for performant pulling down batches of that data while you are doing a large scale training job."Ethan Rosenthal30:58
Who should watch
You are building training infrastructure for video, image, audio, or other large multimodal datasets and need to decide where derived features should live.
Your research and production teams exchange serialized models or scripts, and you want a shared development pattern with reusable code and tests.
You run distributed GPU training and want workers to stream selected batches instead of copying an entire dataset to every machine.
