Machine learning products change with their data and environment, so infrastructure must support rapid iteration and ongoing maintenance.
2
Airflow remains useful for existing data workloads, while compute-heavy machine learning jobs can be handed to a system such as Flight without migrating everything.
3
Flight treats tasks as versioned, independently runnable units that can contain anything from a Python function to a distributed Spark job or model-training workload.
Summary
Ketan Umare explains why machine learning workflows need different infrastructure from traditional software and data pipelines. Models can become invalid when external conditions change, as happened during the pandemic, so teams need to iterate, retry, reuse results, and keep production systems under active care. He argues that Airflow works well for many existing workloads, but its assumptions make distributed training, GPU jobs, checkpointing, feature generation, and large networks of changing pipelines harder to manage. Flight uses versioned tasks that can run independently and can expand from a local Python function to distributed workloads on Kubernetes, Spark, SageMaker, or other systems. Ketan also describes signals for human approval, automated rollout decisions, and model or data drift responses. He separates Flight as open source from Union as the company providing managed deployment, support, and higher-level ML practices.
Machine learning products need infrastructure built for constant change
Ketan says his experience across high-frequency trading, banking, map making, logistics, ride sharing, and cloud repeatedly exposed infrastructure problems that looked like pipelines. Machine learning made the problem more urgent because models depend on assumptions about data and the outside world. He gives the example of models built before the pandemic that stopped working when conditions changed, even though the companies themselves had not fundamentally changed. Teams may need to change data, revise assumptions, and ship new versions quickly. Ketan argues that traditional software infrastructure assumes more stability, while machine learning systems need constant iteration and continued maintenance after deployment.
Airflow is useful when it works, but migration alone brings little value
Ketan takes a practical position on Airflow. Companies that have used it for years already have pipelines, code, and operating habits around it, so rewriting everything in a new tool may offer little benefit when the current system is sufficient. He draws a boundary around compute-intensive machine learning work. Rapid experiments, repeated attempts, reuse of earlier results, feature-generation work, and distributed training create needs that Airflow does not handle naturally. His recommendation is incremental adoption. Teams can keep Airflow for existing workloads and hand specific training or GPU-heavy jobs to Flight through an operator, rather than treating migration as an all-or-nothing decision.
Large machine learning pipeline estates become difficult before teams expect it
Ketan describes a Lyft pricing team with seven or eight people, five production models, and around 600 pipelines working together. Parts of the system were reused, deprecated, revived, or changed, and the team often could not safely delete code because some piece might be needed later. He says this complexity grows quickly as companies add repositories, teams, and dependencies. Spotify-scale organizations can have tens of thousands of pipelines and many repositories containing pipeline code. The practical problem is debugging and changing one part without knowing what depends on it. Teams often start with a small deployment and only discover the maintenance problem after it has expanded.
Flight makes each task independently runnable and versioned
Ketan says Flight keeps familiar pipeline structure while changing the smallest unit of execution. A task can run on its own, which lets a user register it and execute it directly. Tasks are versioned, so teams can run older versions and track which inputs caused earlier versions to run. The task can also have different levels of execution complexity. A simple Python function can run in one container, while the same task model can represent a distributed Spark job or another multi-container workload. Ketan says this keeps the code and user experience consistent while the system handles the infrastructure needed for the selected execution target.
Machine learning orchestration needs checkpointing and model-aware data handling
Ketan says training work should not be forced into many small workflow steps because that can be slow and use the wrong abstraction. Flight supports interrupt checkpointing, including passing a checkpoint across retries without requiring the user to manage that transfer. The system also understands model outputs and can store weights in a way that lets a model move between GPU and CPU environments without extra user work. He gives tensor serialization as another example of a machine learning-specific assumption that can be built into the orchestration layer. Ketan presents these behaviors as extensible parts of the system, so additional support can be added as machine learning tools change.
Flight can manage distributed workloads while existing orchestration remains in place
Ketan describes Flight as a layer over Kubernetes that can bring up and manage resources for workloads such as Ray. A user can write code for a Ray job while Flight creates the cluster, and later workflow steps can reuse the same cluster during the workflow's lifetime. He says this allows teams to keep Airflow for simple or legacy jobs and send selected workloads to Flight. The same pattern can apply to Spark, SageMaker, EMR, or other compute-intensive systems. At Lyft, Airflow operators handed Spark jobs to Flight. Ketan argues that this approach avoids forcing a full rewrite while giving heavier jobs access to the execution model they need.
Signals can connect deployment stages to people or automated checks
Ketan describes a rollout that sends 25 percent of traffic to a new model, waits for a signal, then moves to 50 percent and eventually 100 percent. The signal could come from a person approving the next step, or from automated checks based on production metrics, model drift, data drift, or newly available ground truth. A system could move forward when accuracy improves, or roll back when it degrades. He says Flight is building the underlying workflow and API support for this pattern, with native user interface support planned. The same mechanism could support labeling and semi-supervised learning.
Union adds managed operations and higher-level practices around Flight
Ketan separates the open-source Flight project from Union, the company. Union contributes heavily to Flight and does not claim ownership of its trademark or rights. The company also helps users handle deployment infrastructure, ingress, security, and Kubernetes setup, which can otherwise delay adoption. Union Cloud lets teams use a managed control plane while data remains in their own virtual private cloud. Ketan says Union is also building higher-level machine learning practices, including ways to reduce the difference between training-time feature transformations and production prediction code. He describes Union as an infrastructure partner for teams that want to adopt Flight quickly or move toward cloud-native systems.
"You should be able to change whatever you feel like, try something new, get an output, and potentially that output is not going to work, so try again and try again and try again."Ketan Umare17:20
Who should watch
You are deciding whether to keep Airflow, add an ML-specific orchestrator, or migrate existing workflows.
Your team has growing numbers of model-training pipelines and struggles with dependencies, retries, versioning, or distributed compute.
You need staged model rollouts that can pause for human approval or respond to production metrics and drift signals.
