# Training pipelines and orchestration

A pack of 8 sessions from the MLOps Community YouTube channel, in the order to watch them. 8h 02m of video.
Page: https://mlopstalks.com/packs/training-pipelines-and-orchestration

The preprocessing finished, but training never started. A retry appended the same output twice. A failed final step sent you back through hours of unchanged work. These are execution problems around the model, and a scheduler only solves part of them. Begin with dependencies and the separation between coordinating a job and running its compute. Add failure handling, inspectable artifacts and safe reruns before considering more specialized execution. Flyte's history connects task interfaces to cached results; Spark and Kubeflow make the difference between submitting a job and waiting for its completion explicit. Aurora shows these pieces working together across long training workflows. Finish by examining whether the resulting workflow is understandable to the people who depend on it. The demonstrations illustrate execution decisions, rather than rank today's orchestrators.

## This pack is for you if

- Someone has to watch preprocessing finish before starting the next training job.
- Retries and backfills repeat side effects or rerun expensive work unnecessarily.
- Your pipeline coordinates Python, distributed processing and GPU jobs with different completion rules.

## The talks, in order

### 1. Airflow in MLOps

Simon Darr & Byron Allen, Servian | 53:33 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=7dcUWLrGLMw
Summary: https://mlopstalks.com/talks/airflow-in-mlops.md

Why first: Separate the coordinator from the computation. Darr and Allen use Airflow to order jobs while other systems do the processing, then explain why reruns must not repeatedly append or partially replace results. Their 2020 discussion establishes the execution contract before adding convenience around it.

### 2. Orchestrating Machine Learning Workflows with Prefect

Kevin Kho, Prefect | 1:04:18 | MLOps Meetup
Video: https://www.youtube.com/watch?v=SP6WqCRUkNc
Summary: https://mlopstalks.com/talks/orchestrating-machine-learning-workflows-with-prefect.md

Why here: Retries, timeouts and alternate paths quickly outgrow a simple script. Kho makes those failure decisions visible, then shows dependencies inferred from Python task calls and values. The Orion demo was a 2022 technical preview; its useful question is how much control flow your runtime must discover during execution.

### 3. Argo Workflows

Kemal Tugrul Yesilbek, Beat | 55:15 | MLOps Meetup
Video: https://www.youtube.com/watch?v=yXyHeE8AdYk
Summary: https://mlopstalks.com/talks/argo-workflows.md

Why here: Trying again helps a temporary failure, but cannot repair a misspelled command. Yesilbek demonstrates that distinction and saves artifacts that later steps or an investigator can inspect. This gives retry policy a necessary companion: preserve enough evidence to understand why the attempt failed.

### 4. Journey of Flyte at Lyft and Through Open-source

Ketan Umare, Lyft | 1:05:06 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=jEoUuLZ-lAM
Summary: https://mlopstalks.com/talks/journey-of-flyte-at-lyft-and-through-open-source.md

Why here: Reusing a result requires a system to connect it to the task and inputs that produced it. Umare's account of Flyte separates typed workflow definitions, execution and the data catalog used for caching. Its narrow orchestration scope also clarifies which surrounding ML responsibilities still belong elsewhere.

### 5. Why You Need More Than Airflow

Ketan Umare, Union.ai | 1:11:12 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=JWCNITgLtBU
Summary: https://mlopstalks.com/talks/why-you-need-more-than-airflow.md

Why here: Some training jobs need checkpoint transfer and execution resources that a simple task wrapper does not provide. Umare's later account treats these as reasons to delegate particular workloads, while keeping existing Airflow jobs that already work. Specialized execution can be an addition rather than a wholesale migration.

### 6. Orchestrating Spark Jobs with Kubeflow

Sadik Bakiu, Freelance ML Engineer | 42:58 | MLOps Meetup
Video: https://www.youtube.com/watch?v=ejWRAcFnA8E
Summary: https://mlopstalks.com/talks/orchestrating-spark-jobs-with-kubeflow.md

Why here: Delegating a job introduces a concrete dependency trap: submission is not completion. Bakiu's 2021 demo polls a Spark application's status before allowing the next step, and disables caching for submission and status operations. Read it for that distinction between reusable data and actions that must happen again.

### 7. Team Aurora: Accelerating ML with Kubeflow

Maurizio Vitale, Vinay Anantharaman & Ankit Aggarwal, Aurora | 54:57 | MLOps Meetup
Video: https://www.youtube.com/watch?v=8TIJx7jjL7g
Summary: https://mlopstalks.com/talks/team-aurora-accelerating-ml-with-kubeflow.md

Why here: Aurora combines the earlier decisions in a long-running ML workflow. Build artifacts, orchestration and compute remain separate, while reusable components launch work on batch systems and SageMaker and report its outcome. The case shows how several execution systems can participate in one traceable process.

### 8. Evolving Workflow Orchestration

Alex Milowski, Entrepreneur and Computer Scientist | 1:14:35 | MLOps Podcast
Video: https://www.youtube.com/watch?v=u5y9HF52oVk
Summary: https://mlopstalks.com/talks/evolving-workflow-orchestration.md

Why last: A runnable workflow can still be hard to understand. Milowski examines what gets lost when compact Python replaces an inspectable process description, especially for collaborators who cannot read the implementation. After arranging tasks and failure paths, check whether the people using the result can explain what must happen before each step runs.

## Editor's note

Kho makes failure handling part of the workflow, and Aurora separates orchestration from the systems doing the compute. ZenML lets teams define Python pipeline steps and choose the underlying orchestrator and artifact store through configuration. Runs record their steps, inputs, outputs and code version, providing a trace from a resulting model back to the workflow that produced it.

Written by the MLOps Talks editors (the ZenML team), not by any of the speakers.