# Metaflow: Supercharging Our Data Scientist Productivity

Ravi Kiran Chirravuri, Netflix | MLOps Meetup | Episode 41 | 1:00:30
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=1iuJvF_SE9Q
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/metaflow-supercharging-our-data-scientist-productivity
Published: 2020-11-10
Tags: developer-experience, orchestration

## TL;DR
- Metaflow gives data scientists a single Python or R library for moving from local prototyping to cloud execution and scheduled production workflows.
- The framework stores workflow state and artifacts, versions dependencies, and supports resuming a failed run from a successful checkpoint.
- Metaflow keeps the workflow syntax close to idiomatic Python while making stronger infrastructure choices underneath, including storage, compute, and orchestration integrations.

## Summary
Ravi Kiran Chirravuri explains why Netflix built Metaflow after observing data scientists work through the full machine learning project lifecycle. A useful framework has to cover more than model code. It must handle data access, compute, scheduling, dependencies, state, logs, artifacts, and failures. Metaflow presents these concerns through a human-oriented Python or R library. Users can prototype locally, send selected steps to AWS Batch, express branching and parallel work, and connect workflows to AWS Step Functions. Metaflow stores state and artifacts in S3, tracks runs in a metadata database, and uses content addressing to avoid duplicate stored values. Its resume feature lets users reuse completed steps while iterating on later ones or reproduce a production failure locally. Ravi also explains Netflix's design choices, including simple DAG syntax, static validation, inline resource declarations, and opinionated infrastructure defaults. The talk is candid about limits around input-data versioning and integrations outside AWS.

## Key ideas
### Metaflow began with observing where data scientists lose time
[05:20](https://www.youtube.com/watch?v=1iuJvF_SE9Q&t=320s)
Before building Metaflow, Netflix's machine learning infrastructure team spent several months working alongside data scientists. They studied a typical pipeline and looked for tasks that were difficult or uninteresting to the people building models. The team also examined software engineering problems that appeared during the work. The goal was to abstract those concerns away so data scientists could spend more time on the modeling and feature work. Ravi frames the project around the complete lifecycle, from an idea and a notebook through cloud execution, scheduling, production failures, and iteration.

### A production data science project touches a thick infrastructure stack
[09:40](https://www.youtube.com/watch?v=1iuJvF_SE9Q&t=580s)
Ravi describes a project moving through data storage, modeling code, compute, scheduling, software architecture, versioning, deployment, monitoring, and feature development. Data may come from files, a database, or a multi-petabyte data lake. Compute can range from a laptop to a large container system. The data scientist's interests tend to concentrate near the top of this stack, while infrastructure teams have more opinions and responsibilities at the lower layers. Metaflow packages those lower-level choices into one library while leaving users freedom at the higher level. The open-source project supports Python and R and includes AWS integrations.

### Metaflow uses simple DAG syntax and idiomatic Python
[14:09](https://www.youtube.com/watch?v=1iuJvF_SE9Q&t=849s)
Metaflow treats a machine learning workflow as a directed acyclic graph and aims to let users express it close to how they think about the work. Steps use an @step decorator, transitions use self.next, and workflows have start and end steps. Static branches must eventually join, and the join step can access the inputs from the branches. Static validation catches some mistakes before cloud execution, which helps during rapid prototyping. Ravi says the framework balances sensible defaults with explicit user intent. A training step can declare resource needs such as 16 CPUs, keeping that information visible in the workflow rather than in a separate configuration location.

### Users can move selected steps from a laptop to AWS Batch
[18:56](https://www.youtube.com/watch?v=1iuJvF_SE9Q&t=1136s)
Metaflow is cloud-first, but it preserves a local prototyping experience. A user can run some steps on a laptop and send others to AWS Batch when they need more CPU or memory. Ravi gives the example of a CPU-heavy step and a data-manipulation step that needs enough memory to load a large data frame. The user pays for cloud compute only when it is used. For workloads that need multiple boxes, the foreach construct creates copies of a step with distinct inputs. This supports patterns such as running a hyperparameter grid across local processes or AWS containers, followed by a join step.

### Metaflow treats failures as a normal part of workflow operation
[23:03](https://www.youtube.com/watch?v=1iuJvF_SE9Q&t=1383s)
Metaflow persists workflow state and data on storage backed by AWS S3 so users can reproduce a production failure on a laptop. Values stored under self are versioned and placed in the data store. Metaflow compresses them and uses content addressing, which avoids duplicate copies when the value has not changed across tasks. The framework also keeps logs and task artifacts available through its Python client. Users can inspect successful outputs, standard error logs, and run history, then build notebooks or dashboards from that information. Ravi's design goal is to make failure recovery part of the normal workflow rather than an afterthought.

### Dependency management aims to make environments reproducible
[27:10](https://www.youtube.com/watch?v=1iuJvF_SE9Q&t=1630s)
Metaflow uses Anaconda's offering to create isolated environments with reproducible versions. A user can specify the packages needed by a step and test an alternate version of a library in another step. Ravi gives TensorFlow 1.14 and TensorFlow 2.0 as an example of comparing environments when an API change might cause compatibility problems. Metaflow maps the short dependency specification to a packaged environment for remote execution and freezes transitive dependencies. Ravi qualifies the reproducibility claim because input data usually comes from a data lake and may be too large to version alongside the pipeline. Teams can follow practices such as keeping immutable data copies in S3.

### The Python client exposes runs, artifacts, and namespaces
[30:50](https://www.youtube.com/watch?v=1iuJvF_SE9Q&t=1850s)
Metaflow versions information about flows and runs and provides a Python client for inspecting it. Namespaces let different users run the same flow without overwriting one another's results. They are guardrails rather than isolated silos, since users can still inspect the global namespace and collaborate. The same client can support monitoring notebooks and business-facing dashboards. Ravi describes using it to inspect a run's values, logs, and artifacts, then building a view such as accuracy across recent runs. Metaflow also collects logs from subprocesses and remote cloud machines so users retain visibility after a task completes or fails.

### Metaflow integrates with schedulers instead of replacing them
[35:27](https://www.youtube.com/watch?v=1iuJvF_SE9Q&t=2127s)
Ravi says Metaflow is not another DAG scheduler. The framework separates workflow intent from orchestration and can translate a Metaflow DAG into a scheduler's specification. Its open-source reference integration uses AWS Step Functions for scheduling and AWS Batch for compute. Step Functions decides when tasks run and how much parallelism to use, while AWS Batch runs the individual tasks. This separation means a user can keep the same workflow definition while moving from local execution to a production scheduler. Ravi describes scheduler integrations as a way to hide the different DSLs and infrastructure details that would otherwise reach the data scientist.

### Resume reuses completed work while users fix later steps
[38:16](https://www.youtube.com/watch?v=1iuJvF_SE9Q&t=2296s)
The resume feature starts from an existing run and reuses successful portions of it. This is useful when data manipulation takes a long time and the user wants to try several training approaches without repeating the earlier work. It also supports a production-debugging loop: reproduce a failed run locally, make and test a fix, then deploy the corrected workflow again. Because Metaflow stores state, artifacts, and metadata for each run, it can identify the completed work and make it available to the resumed execution. Ravi presents this as a bridge between local development and production execution.

## Notable quotes
- Ravi Kiran Chirravuri: "Metaflow is a human-centric framework for building and managing real-life data science projects." (08:48)
- Ravi Kiran Chirravuri: "We just want to store everything." (26:02)
- Ravi Kiran Chirravuri: "Failures are a feature and not an afterthought." (23:03)
- Ravi Kiran Chirravuri: "Dependency management is hard and we want to get as close to reproducibility of your workflows as possible." (29:38)
- Ravi Kiran Chirravuri: "I think our get started should be as simple as pip install metaflow on your laptop." (55:54)

## Tools & references mentioned
- Metaflow
- Netflix
- AWS
- AWS Batch
- AWS Step Functions
- Amazon S3
- AWS RDS
- Anaconda
- Python
- R
- Jupyter notebooks
- TensorFlow

## Who should watch
- You are a data scientist who wants to develop locally and send only resource-heavy workflow steps to the cloud.
- Your team needs to preserve run state, inspect artifacts and logs, or reproduce a production failure before deploying a fix.
- You are building machine learning infrastructure and want examples of how to hide scheduler and storage complexity behind a simple workflow API.

## Editor's note

Ravi Kiran Chirravuri describes how Metaflow handles production failures by storing workflow state and artifacts so a run can be reproduced locally and resumed from completed steps. ZenML records each run's steps, inputs, outputs, and code version, so a model or artifact can be traced to the data and code that produced it. Unchanged steps are cached rather than recomputed.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Human-centric ML Infrastructure: A Netflix Original](https://mlopstalks.com/talks/human-centric-ml-infrastructure-a-netflix-original) (Savin Goyal, Netflix, 56:07)
- [Comparing ZenML, Metaflow, and all the other DAG tools](https://mlopstalks.com/talks/comparing-zenml-metaflow-and-all-the-other-dag-tools) (, 55:39)
- [Making MLflow](https://mlopstalks.com/talks/making-mlflow) (Corey Zumar, Databricks, 59:11)
- [Airflow Sucks for MLOps](https://mlopstalks.com/talks/airflow-sucks-for-mlops) (Stephen Bailey, Whatnot, 1:05:53)
- [Orchestrating Machine Learning Workflows with Prefect](https://mlopstalks.com/talks/orchestrating-machine-learning-workflows-with-prefect) (Kevin Kho, Prefect, 1:04:18)