# Human-centric ML Infrastructure: A Netflix Original

Savin Goyal, Netflix | MLOps Meetup | Episode 44 | 56:07
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=TzRNZO2E-eM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/human-centric-ml-infrastructure-a-netflix-original
Published: 2020-12-14
Tags: debugging, experiment-tracking, orchestration, platform-teams

## TL;DR
- Metaflow gives Netflix data scientists freedom to choose their algorithms and tools while handling shared concerns such as lineage, experiment tracking, scaling, and reproducibility.
- Savin Goyal describes reproducibility as a way to debug failed production workflows by recreating the same code, data, dependencies, runtime, and model randomness.
- Metaflow reduces operational work by letting users move from laptops to larger compute instances with small code changes, while queues and priorities limit resource use.

## Summary
Savin Goyal explains why Netflix built Metaflow for data scientists working across content analysis, fraud detection, recommendation systems, and other machine learning problems. The framework lets users work in Python and R without forcing a particular modeling library or workflow language. It handles recurring infrastructure work such as code snapshots, lineage, experiment tracking, data movement, scheduling, and execution on different compute environments. Goyal frames the design around reproducibility, scalability, and usability. Reproducibility helps teams recreate failures and inspect how data, code, dependencies, infrastructure, and randomness affected a run. Scalability lets a user move from a laptop to a larger instance, or run parameterized jobs in parallel, without managing containers and data transfers directly. Usability means the system should work for people rather than turning data scientists into systems engineers. He also discusses queues, access policies, Metaflow's open-source integrations, and adoption across several industries.

## Key ideas
### Metaflow gives data scientists freedom while handling shared ML concerns
[02:38](https://www.youtube.com/watch?v=TzRNZO2E-eM&t=158s)
Metaflow was built for Netflix's varied machine learning work, including content analysis, fraud detection, and intelligent infrastructure. Goyal says Netflix data scientists are experts in machine learning, statistics, and related fields, so they should be able to choose their algorithms, monitoring tools, and modeling approach. The framework avoids imposing one way to do data science. It still handles concerns that appear across projects, such as lineage, experiment tracking, training and deployment at scale, container launches, reproducibility, and moving large data sets between systems. Netflix built the framework by addressing individual pain points over time, until the collection of tools became a coherent system.

### Metaflow connects data, compute, containers, scheduling, and ML operations
[05:54](https://www.youtube.com/watch?v=TzRNZO2E-eM&t=354s)
Goyal places Metaflow within a wider stack. Netflix stores data and metadata in S3, then gives users several compute choices, from laptops and cloud workstations to EC2 instances with unusual resource combinations. Containers run on Netflix's Titus system, and workflow schedulers trigger training when new data arrives or on a regular cadence. Machine learning adds concerns such as repeatability, experimentation, and data scientist productivity. Metaflow links these layers so a user can launch many jobs, move data to the execution environment, collect results, and preserve lineage without handling each systems task manually. He says the cost of lost data scientist time matters alongside infrastructure spending.

### Runway manages serving models while Metaflow focuses on training
[13:27](https://www.youtube.com/watch?v=TzRNZO2E-eM&t=807s)
Netflix has a long history with recommendation infrastructure built around the JVM and Scala ecosystem. Goyal describes Runway as a model management system for complicated recommendation models that serve inferences to Netflix members. Runway tracks which models are in use and how they perform, with internal monitoring attached to that lifecycle. Metaflow addresses different work, mainly training models in Python and R and hosting some models for internal use cases through a function-as-a-service platform. The separation reflects different needs between managing member-facing inference and supporting data scientists during model development and training.

### Data access needs to avoid unnecessary query-engine bottlenecks
[15:45](https://www.youtube.com/watch?v=TzRNZO2E-eM&t=945s)
Netflix keeps its data in S3 and uses engines such as Presto and Spark to query it. Goyal says data scientists often need one table partition or a small group of partitions rather than a large analytical query. Sending those requests through a shared query engine can be inefficient and can make users wait behind heavy queries. His team built tooling that accesses warehouse data directly and bypasses those engines for this pattern. The wider data platform also handles cataloging, governance, and helping users discover available data assets. Metaflow relies on that platform rather than replacing it.

### Reproducibility is a practical way to recreate failures
[29:10](https://www.youtube.com/watch?v=TzRNZO2E-eM&t=1750s)
Goyal rejects a narrow definition in which a data set and source code are enough. Reproducing a run may require an audit log that allows the system to return to a prior state, along with versioned data, user code, library dependencies, runtime infrastructure, and model randomness. Each part brings costs, so teams must decide which guarantees matter for a particular workload. At Netflix, the immediate value is debugging. If a production training workflow fails, a user should be able to run the same pipeline and recreate the error. Without that ability, fixing the problem becomes guesswork. Metaflow snapshots the code for every execution and stores it for later use.

### Scaling should often start with a larger instance before distributed training
[33:14](https://www.youtube.com/watch?v=TzRNZO2E-eM&t=1994s)
Data scientists commonly begin with a data set that fits on a laptop or cloud workstation. When the full data set no longer fits, Goyal prefers a low-overhead step first: move the same code to a larger instance. In Metaflow, a user can change an annotation after an out-of-memory failure and run on an instance with more memory, while the system handles the remote execution and returns the results. The same primitives can run several parameterized copies on separate instances. Distributed training remains available when model or data size requires it, but vertical scaling and simple data sharding can solve many problems earlier.

### Queues and priorities keep flexible compute from becoming uncontrolled spending
[36:24](https://www.youtube.com/watch?v=TzRNZO2E-eM&t=2184s)
Metaflow can use container orchestration systems such as Titus and AWS Batch, which provide queues and priorities. An exploratory queue can have a fixed amount of CPU or GPU capacity, so additional jobs wait instead of consuming unlimited resources. Once work is approved for production, it can move to a queue with more capacity. This gives data scientists room to experiment while allowing the organization to control how much compute each class of work can use. The policy comes from the compute environment and its access rules rather than from a hard-coded Metaflow limit.

### Usability means hiding operations without hiding what happened
[37:43](https://www.youtube.com/watch?v=TzRNZO2E-eM&t=2263s)
Goyal says tools should work for people, so Metaflow is a Python library rather than a domain-specific language. Data scientists can use languages they already work with instead of learning a separate workflow syntax. The system removes tasks such as building Docker images, reading orchestration manuals, interpreting infrastructure errors, and moving data by hand. It still exposes enough information for users to understand failures. Metaflow captures the code and runtime for each execution, which avoids requiring users to record Git state and other details themselves. Goyal's usability goal is to remove extra actions around operations while preserving the information needed to diagnose a run.

## Notable quotes
- Savin Goyal: "My fundamental belief is that tools should work for people and people should not work for tools." (37:43)
- Savin Goyal: "There is no one true system that just promises you the holy grail of reproducibility." (31:04)
- Savin Goyal: "As a data scientist I don't really want to know all of that. I just want the capability to just here is my code, the system should know everything else, and the compute should just execute and give me back the results." (34:46)
- Savin Goyal: "Things are going to break, things are going to fail, your data distributions are going to change." (32:12)

## Tools & references mentioned
- Netflix
- Metaflow
- Runway
- S3
- EC2
- Docker
- Titus
- Kubernetes
- AWS Batch
- Airflow
- Presto
- Spark
- Python
- R
- TensorFlow
- Kayenta
- Pinnacle
- MLflow
- Cortex

## Who should watch
- You are building an internal ML platform and need to decide which infrastructure work belongs in a framework and which belongs in existing data or compute systems.
- Your data scientists spend time rebuilding environments, moving data, or diagnosing workflows that cannot be rerun after a failure.
- You need a practical discussion of how an open-source ML framework can fit with existing cloud, model tracking, and model serving tools.

## Editor's note

Savin Goyal says reproducibility at Netflix means recreating a failed production workflow with its code, data, dependencies, runtime, and model randomness. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a model or artifact can be traced to the data and code that produced it. Its configurable stack also lets the same pipeline run across different infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Metaflow: Supercharging Our Data Scientist Productivity](https://mlopstalks.com/talks/metaflow-supercharging-our-data-scientist-productivity) (Ravi Kiran Chirravuri, Netflix, 1:00:30)
- [Making MLflow](https://mlopstalks.com/talks/making-mlflow) (Corey Zumar, Databricks, 59:11)
- [Packaging MLOps Tech Neatly for Engineers and Non-engineers](https://mlopstalks.com/talks/packaging-mlops-tech-neatly-for-engineers-and-non-engineers) (Jukka Remes, Haaga-Helia University of Applied Sciences, 8wave AI, 55:31)
- [The Post Modern Stack](https://mlopstalks.com/talks/the-post-modern-stack) (Jacopo Tagliabue, Coveo, 1:04:58)
- [MLOps Engineering Labs Recap, Part 2](https://mlopstalks.com/talks/mlops-engineering-labs-recap-part-2) (Laszlo Sranger & Artem Yushkovsky, Neuro & Paulo Maia, Nilgai, 1:04:16)