Flyte grew from Lyft's need to run many kinds of workflows while hiding infrastructure details from engineers.
2
Flyte focuses on workflow orchestration and integrates with systems such as Kubernetes, Spark, Databricks, and SageMaker instead of trying to provide every ML platform function.
3
Ketan Umare says Flyte's open-source direction depends on a community because the product's ambition is too broad for one team to build alone.
Summary
Ketan Umare explains how Flyte developed inside Lyft from an effort to automate ETA model pipelines into a general workflow and infrastructure orchestration system. Lyft has teams working on marketplace systems, mapping, vision, autonomous driving, fraud, growth, and analytics, with workloads ranging from small model training jobs to large distributed processing. Flyte runs workflows and coordinates the infrastructure underneath them, while relying on Kubernetes and external systems for specific compute services. Its users define typed, reusable tasks and workflows, then submit them without managing machines. The system records execution history, supports caching, and tracks data lineage. Ketan also describes Flyte's open-source lessons. The team narrowed the product to orchestration, built extensibility into the backend, and learned from users such as Spotify. He is direct about the remaining installation and usability problems. His broader goal is to let engineers share solved workflow and infrastructure components in the same way developers share code on GitHub.
Ketan Umare came to ML infrastructure through data scale and distributed systems
Ketan says he did not begin with a machine learning background. While working on banking and anti-money-laundering systems, he found that the algorithms were manageable, but the data volume and speed requirements were difficult. That led him toward distributed systems and high-performance computing at Georgia Tech. He later worked on CUDA, high-frequency trading, Amazon logistics and mapping, Oracle's Bare Metal Cloud, and Lyft's mapping and infrastructure. His interest in Flyte follows the same pattern: make large workloads efficient, quick, and deployable in production.
Lyft's ML infrastructure has to support many different workload shapes
Ketan describes Lyft's requirements as a wide range rather than a single standard ML use case. Marketplace systems cover locations, ingested data, ETA, dispatch, pricing, and forecasting. Mapping involves live feeds and sensor data, while autonomous driving creates large offline workloads. Some teams train and deploy a simple model from 100 megabytes of data. Others process terabytes with distributed models that may end up in a car rather than a service. This variety means the infrastructure has to support both straightforward jobs and complex data-processing pipelines.
Flyte hides machines while coordinating workflows and their underlying compute
Ketan says Flyte is a workflow engine combined with orchestration of the low-level infrastructure needed to run each step. At Lyft, around 600 engineers were actively using it weekly for Spark jobs, feature engineering, ETL-style processing, analytics, model retraining, log aggregation, security checks, financial analysis, executive dashboards, and autonomous-driving data work. Users write code and press a button. They are paged about failures in their code rather than machine or networking problems. Flyte itself does not replace Kubernetes or every external compute service. It coordinates with those systems.
Flyte grew from an internal Airflow prototype into a Kubernetes-based platform
Ketan began with ETA models and a traffic model that combined Spark, XGBoost, Python validation, and other steps. Engineers initially ran pieces from laptops or through existing systems. He automated the process with Airflow, then built an interface on top of it as more teams asked to reuse the workflow. A later version used AWS Step Functions, AWS Batch, and Kubernetes. As usage grew, the team invested in a Kubernetes design, using Go for the backend and Python for the user SDK. The interface stayed consistent while the implementation changed underneath it.
Flyte keeps its scope narrow and uses extensions for the rest
Ketan draws a firm boundary around Flyte. It focuses on orchestration. It does not provide model serving or notebook hosting, and its tracking covers what runs inside Flyte rather than arbitrary experiments elsewhere. The system can connect to Kubernetes operators, Databricks, SageMaker, and other services through extensions. Backend extensions are managed by the Flyte team, while users can add their own extensions without sharing them. Ketan says this division helps Flyte concentrate its effort on one large problem instead of attempting to build an entire ML platform.
Community users expose workflow problems that one team would miss
Ketan calls his first open-source project humbling and says he is grateful for every contribution. Spotify showed the team a gap in Flyte's task sharing model. A platform team might publish reusable tasks, but updating a task's interface can break workflows that depend on it. Flyte did not yet support schema evolution for that case, so the feedback became a planned feature. Ketan believes a project with Flyte's scope needs a community. The aim is for people to build and share data and ML workflow components that others can use without rebuilding the infrastructure.
Reactive pipelines should run when their data is ready
Ketan describes reactive pipelines as a response to the limitations of time-based workflow scheduling. An Airflow sensor can wait for an S3 file or partition, but the workflow still assumes that the file should appear at a particular time and needs timeout rules. In Flyte's proposed model, the event is the contract. A pipeline would run when a required artifact appears, without exposing the underlying publish-subscribe machinery to the user. The same model could connect pipelines through shared artifacts, rebuild an artifact, or trigger an alternate pipeline after a failure.
The architecture separates workflow definition, execution, and lineage
Ketan outlines five main components. Users install the Flyte Kit SDK, which defines tasks and workflows. Flyte Admin provides the gRPC and REST-facing service, stores workflow and execution information, and supports the console and CLI. During compilation, Flyte checks typed inputs, outputs, and parameters so some pipeline errors appear early. Flyte Propeller converts a run into Kubernetes custom resources and executes the workflow through plugins. Data Catalog records outputs and lineage, then allows Flyte to reuse cached results. If one upstream task is fixed, downstream work can avoid rerunning unchanged steps.
"What we want to solve to be honest as a community is make it easy for anybody who's writing any sort of software anywhere in the world, hopefully in the data and ML space, to not think about infrastructure ever again."37:39
Who should watch
You run batch ML or data workflows and want users to submit code without operating the underlying machines.
Your team is comparing workflow orchestration with broader ML platform products and needs a clear account of what Flyte does not include.
You are building an open-source infrastructure project and want practical lessons about scope, extensibility, installation, and feedback from early users.
