# Journey of Flyte at Lyft and Through Open-source

Ketan Umare, Lyft | MLOps Coffee Sessions | Episode 12 | 1:05:06

Source: https://www.youtube.com/watch?v=jEoUuLZ-lAM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/journey-of-flyte-at-lyft-and-through-open-source
Published: 2020-10-10
Tags: caching, data-engineering, open-source, orchestration

## TL;DR
- Flyte grew from Lyft's need to run many kinds of workflows while hiding infrastructure details from engineers.
- Flyte focuses on workflow orchestration and integrates with systems such as Kubernetes, Spark, Databricks, and SageMaker instead of trying to provide every ML platform function.
- Ketan Umare says Flyte's open-source direction depends on a community because the product's ambition is too broad for one team to build alone.

## Summary
Ketan Umare explains how Flyte developed inside Lyft from an effort to automate ETA model pipelines into a general workflow and infrastructure orchestration system. Lyft has teams working on marketplace systems, mapping, vision, autonomous driving, fraud, growth, and analytics, with workloads ranging from small model training jobs to large distributed processing. Flyte runs workflows and coordinates the infrastructure underneath them, while relying on Kubernetes and external systems for specific compute services. Its users define typed, reusable tasks and workflows, then submit them without managing machines. The system records execution history, supports caching, and tracks data lineage. Ketan also describes Flyte's open-source lessons. The team narrowed the product to orchestration, built extensibility into the backend, and learned from users such as Spotify. He is direct about the remaining installation and usability problems. His broader goal is to let engineers share solved workflow and infrastructure components in the same way developers share code on GitHub.

## Key ideas
### Ketan Umare came to ML infrastructure through data scale and distributed systems
[01:50](https://www.youtube.com/watch?v=jEoUuLZ-lAM&t=110s)
Ketan says he did not begin with a machine learning background. While working on banking and anti-money-laundering systems, he found that the algorithms were manageable, but the data volume and speed requirements were difficult. That led him toward distributed systems and high-performance computing at Georgia Tech. He later worked on CUDA, high-frequency trading, Amazon logistics and mapping, Oracle's Bare Metal Cloud, and Lyft's mapping and infrastructure. His interest in Flyte follows the same pattern: make large workloads efficient, quick, and deployable in production.

### Lyft's ML infrastructure has to support many different workload shapes
[10:04](https://www.youtube.com/watch?v=jEoUuLZ-lAM&t=604s)
Ketan describes Lyft's requirements as a wide range rather than a single standard ML use case. Marketplace systems cover locations, ingested data, ETA, dispatch, pricing, and forecasting. Mapping involves live feeds and sensor data, while autonomous driving creates large offline workloads. Some teams train and deploy a simple model from 100 megabytes of data. Others process terabytes with distributed models that may end up in a car rather than a service. This variety means the infrastructure has to support both straightforward jobs and complex data-processing pipelines.

### Flyte hides machines while coordinating workflows and their underlying compute
[20:15](https://www.youtube.com/watch?v=jEoUuLZ-lAM&t=1215s)
Ketan says Flyte is a workflow engine combined with orchestration of the low-level infrastructure needed to run each step. At Lyft, around 600 engineers were actively using it weekly for Spark jobs, feature engineering, ETL-style processing, analytics, model retraining, log aggregation, security checks, financial analysis, executive dashboards, and autonomous-driving data work. Users write code and press a button. They are paged about failures in their code rather than machine or networking problems. Flyte itself does not replace Kubernetes or every external compute service. It coordinates with those systems.

### Flyte grew from an internal Airflow prototype into a Kubernetes-based platform
[24:08](https://www.youtube.com/watch?v=jEoUuLZ-lAM&t=1448s)
Ketan began with ETA models and a traffic model that combined Spark, XGBoost, Python validation, and other steps. Engineers initially ran pieces from laptops or through existing systems. He automated the process with Airflow, then built an interface on top of it as more teams asked to reuse the workflow. A later version used AWS Step Functions, AWS Batch, and Kubernetes. As usage grew, the team invested in a Kubernetes design, using Go for the backend and Python for the user SDK. The interface stayed consistent while the implementation changed underneath it.

### Flyte keeps its scope narrow and uses extensions for the rest
[38:40](https://www.youtube.com/watch?v=jEoUuLZ-lAM&t=2320s)
Ketan draws a firm boundary around Flyte. It focuses on orchestration. It does not provide model serving or notebook hosting, and its tracking covers what runs inside Flyte rather than arbitrary experiments elsewhere. The system can connect to Kubernetes operators, Databricks, SageMaker, and other services through extensions. Backend extensions are managed by the Flyte team, while users can add their own extensions without sharing them. Ketan says this division helps Flyte concentrate its effort on one large problem instead of attempting to build an entire ML platform.

### Community users expose workflow problems that one team would miss
[34:42](https://www.youtube.com/watch?v=jEoUuLZ-lAM&t=2082s)
Ketan calls his first open-source project humbling and says he is grateful for every contribution. Spotify showed the team a gap in Flyte's task sharing model. A platform team might publish reusable tasks, but updating a task's interface can break workflows that depend on it. Flyte did not yet support schema evolution for that case, so the feedback became a planned feature. Ketan believes a project with Flyte's scope needs a community. The aim is for people to build and share data and ML workflow components that others can use without rebuilding the infrastructure.

### Reactive pipelines should run when their data is ready
[42:14](https://www.youtube.com/watch?v=jEoUuLZ-lAM&t=2534s)
Ketan describes reactive pipelines as a response to the limitations of time-based workflow scheduling. An Airflow sensor can wait for an S3 file or partition, but the workflow still assumes that the file should appear at a particular time and needs timeout rules. In Flyte's proposed model, the event is the contract. A pipeline would run when a required artifact appears, without exposing the underlying publish-subscribe machinery to the user. The same model could connect pipelines through shared artifacts, rebuild an artifact, or trigger an alternate pipeline after a failure.

### The architecture separates workflow definition, execution, and lineage
[51:44](https://www.youtube.com/watch?v=jEoUuLZ-lAM&t=3104s)
Ketan outlines five main components. Users install the Flyte Kit SDK, which defines tasks and workflows. Flyte Admin provides the gRPC and REST-facing service, stores workflow and execution information, and supports the console and CLI. During compilation, Flyte checks typed inputs, outputs, and parameters so some pipeline errors appear early. Flyte Propeller converts a run into Kubernetes custom resources and executes the workflow through plugins. Data Catalog records outputs and lineage, then allows Flyte to reuse cached results. If one upstream task is fixed, downstream work can avoid rerunning unchanged steps.

## Notable quotes
- "The problem was how do I make this efficient and how do I make this quick and really deployable for production." (03:18)
- "They don't think about machines, they write code and hit a button and it scales to their need somewhere they don't even know." (20:41)
- "We solve problems that affect the users first, not just because we think it's cool." (33:45)
- "What we want to solve to be honest as a community is make it easy for anybody who's writing any sort of software anywhere in the world, hopefully in the data and ML space, to not think about infrastructure ever again." (37:39)
- "It is not a machine learning platform. It doesn't have serving." (1:04:32)

## Tools & references mentioned
- MLOps Community
- Demetrios Brinkmann
- TWIML
- Software Engineering Daily
- Georgia Tech
- CUDA
- Amazon
- Oracle Bare Metal Cloud
- Lyft
- Airbnb
- Uber
- AWS
- Envoy
- Kubernetes
- Spark
- SageMaker
- Databricks
- Airflow
- XGBoost
- Step Functions
- Netflix
- Spotify
- Google
- Kubeflow
- TensorFlow Operator
- Flyte Kit
- Flyte Kit Java
- Flyte Admin
- Flyte Console
- Flyte Propeller
- Data Catalog
- Great Expectations
- Weights & Biases

## Who should watch
- You run batch ML or data workflows and want users to submit code without operating the underlying machines.
- Your team is comparing workflow orchestration with broader ML platform products and needs a clear account of what Flyte does not include.
- You are building an open-source infrastructure project and want practical lessons about scope, extensibility, installation, and feedback from early users.

## Editor's note

Ketan Umare says Flyte lets engineers submit workflows without managing machines, while the platform coordinates the infrastructure underneath each step. ZenML lets teams write workflows as Python steps and choose the orchestrator and artifact store through configuration, so the same pipeline code can run on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud service.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLOps EngineeringLabs](https://mlopstalks.com/talks/mlops-engineeringlabs) (Niels Bantilan & Haytham Abuelfutuh, Union.Ai, 1:15:40)
- [MLOps vs ML Orchestration](https://mlopstalks.com/talks/mlops-vs-ml-orchestration) (Ketan Umare, Union.ai, 49:46)
- [Why You Need More Than Airflow](https://mlopstalks.com/talks/why-you-need-more-than-airflow) (Ketan Umare, Union.ai, 1:11:12)
- [The Evolution of Lyft's Feature Store](https://mlopstalks.com/talks/the-evolution-of-lyfts-feature-store) (Devon Mittow, Lyft, 13:13)
- [Scaling MLOps for Computer Vision](https://mlopstalks.com/talks/scaling-mlops-for-computer-vision) (David Espejo, Union & Fabio Grätz, Recogni & Arno Hollosi, Blackshark.ai, 58:54)