# Airflow in MLOps

Simon Darr & Byron Allen, Servian | MLOps Coffee Sessions | Episode 5 | 53:33

Source: https://www.youtube.com/watch?v=7dcUWLrGLMw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/airflow-in-mlops
Published: 2020-08-03
Tags: data-engineering, orchestration

## TL;DR
- Simon Darr describes Airflow as a batch scheduling and orchestration tool that is especially well suited to ETL workflows.
- Byron Allen explains how his team uses Airflow to coordinate machine learning training and batch inference while Kubernetes handles execution environments.
- The guests say Airflow is flexible and easy to extend, but its scheduler, dynamic DAG behavior, Kubernetes integration, and operational complexity need careful handling.

## Summary
Simon Darr and Byron Allen explain where Airflow fits in data engineering and MLOps. Darr describes it as a batch ETL scheduler and says Airflow should coordinate work rather than perform heavy data processing itself. Jobs such as SQL transformations, Spark processing, or model training should run in the systems designed for them. Allen describes a machine learning setup where Airflow coordinates several training and batch inference jobs, while Kubernetes pods provide the runtime resources. The conversation covers Airflow's scheduler, web server, workers, database, executors, and optional message brokers such as Redis or RabbitMQ. A demo shows a DAG written in Python, Snowflake operators, custom operators, Jinja templating, MLflow integration, logging, backfills, and idempotent pipelines. The guests are positive about Airflow's extensibility and community, while being direct about its limits. It is not native to Kubernetes, dynamic DAGs can cause problems, and large deployments have many moving parts.

## Key ideas
### Airflow coordinates batch work instead of doing the heavy processing
[05:50](https://www.youtube.com/watch?v=7dcUWLrGLMw&t=350s)
Simon Darr describes Airflow as a batch scheduling tool and compares it to cron with more capability. His data engineering use case is scheduling SQL scripts and defining dependencies between them. Airflow is the orchestrator, while transformations run in another service, such as a database or Spark. He warns against doing large Pandas or other processing workloads inside an Airflow worker. The tool should trigger and order those jobs, then record their progress. This distinction matters in MLOps because Airflow can coordinate a machine learning workflow without being the system that trains the model or performs the main data processing.

### Machine learning teams can use Airflow to coordinate separate execution systems
[09:06](https://www.youtube.com/watch?v=7dcUWLrGLMw&t=546s)
Byron Allen explains that his team runs several machine learning models in a single training run and then runs batch prediction jobs before promoting the results to model endpoints. Airflow schedules and coordinates those steps. The actual work runs through the Kubernetes Pod Operator, which keeps workflow orchestration separate from the runtime environment and its resource needs. The workflow may trigger training, batch inference, artifact uploads, and simple filesystem operations. Other tools handle parts of the work, including Kubeflow and Polyaxon for some training and tuning tasks. Airflow remains the coordinator rather than the complete ML platform.

### Airflow's architecture has several components and executor choices
[14:44](https://www.youtube.com/watch?v=7dcUWLrGLMw&t=884s)
The discussion breaks Airflow into a web server, scheduler, workers, and a database such as Postgres. A message broker such as Redis or RabbitMQ can help manage long-running tasks and distribute work. The sequential executor is useful for testing and debugging. The local executor can run tasks in parallel on one machine, while Celery can distribute tasks across workers. The Kubernetes executor creates a new pod for each task. These choices affect how much infrastructure a team must operate and how tasks are isolated. The speakers describe the architecture as useful, but admit that the number of moving parts can make Airflow difficult to manage.

### Python-defined DAGs make Airflow easy to extend and parameterize
[19:28](https://www.youtube.com/watch?v=7dcUWLrGLMw&t=1168s)
In the demo, Simon shows a DAG written as Python code. It loads data from Snowflake, forecasts credit or storage spend with Facebook Prophet, and saves a model to MLflow. Airflow operators can use Jinja templating, including the execution date, to parameterize SQL and file paths. Simon also shows how a YAML file could provide a changing set of SQL statements, with Python loops creating the required operators for each run. Custom operators inherit from the base operator and implement the execute method. This lets a team package its own Snowflake-to-CSV behavior and reuse it across pipelines.

### Logs and rendered SQL make failures easier to diagnose
[30:53](https://www.youtube.com/watch?v=7dcUWLrGLMw&t=1853s)
The Airflow interface exposes task status, logs, and the SQL that was actually rendered and executed. Byron explains that logs are the first place to look when something fails. They also help teams understand transformations and investigate data lineage questions, even though Airflow itself is not a data lineage tool. Simon describes building custom operators that record useful details, such as query results, in the logs. Byron says this became especially important when his team had separate Airflow deployments for feature branches and struggled to find logs. They later moved to one Airflow instance with multiple DAGs.

### Airflow has limits around dynamic DAGs, scheduling frequency, and Kubernetes
[22:22](https://www.youtube.com/watch?v=7dcUWLrGLMw&t=1342s)
Simon identifies dynamically changing DAG structure as a major problem. A DAG can be generated from configuration, but changing its structure while it is running creates trouble. He also says Airflow is a batch tool and is a poor fit for very frequent scheduling, since the scheduler evaluates work on an interval. Byron adds that Airflow is not native to Kubernetes, so teams need the Kubernetes Pod Operator or Kubernetes executor to integrate it with a Kubernetes-based ML system. These workarounds can be effective, but teams should consider whether another tool fits the execution environment more directly.

### Idempotency and reproducibility make Airflow workflows safer to rerun
[43:38](https://www.youtube.com/watch?v=7dcUWLrGLMw&t=2618s)
Byron connects Airflow's data engineering design to machine learning workflow needs. An idempotent pipeline produces the same result when run repeatedly, which makes retries and backfills safer. Simon explains the data warehouse version of this idea: a rerun should not keep appending or mutating final tables in ways that make historical recomputation difficult. In the demo, Simon first deletes data for the current run before loading it again, and he recommends using a transaction so users do not see an empty or partially updated result. The speakers also acknowledge that model training can contain randomness, so full determinism is not automatic.

### Airflow's community and operator model give teams room to adapt it
[48:05](https://www.youtube.com/watch?v=7dcUWLrGLMw&t=2885s)
Simon and Byron describe Airflow as a growing open source project with active contributions and a useful operator model. Teams can use built-in operators, Python and Bash operators, or create custom operators when the available ones do not fit. Simon's first open source contribution was a Snowflake-to-Slack operator. The guests say the community was receptive to new contributions, and a team can fork Airflow or build abstractions around it when needed. They also caution against tying an organization too closely to one orchestrator. Data scientists may need help from data engineers to understand Airflow's behavior and avoid its less obvious traps.

## Notable quotes
- Simon Darr: "It's a batch scheduling tool. That's what it is. It's cron on steroids." (05:50)
- Simon Darr: "Airflow is the orchestrator and then you go and run your ETL jobs." (06:30)
- Byron Allen: "What it's really doing is just orchestrating all of these steps." (11:04)
- Simon Darr: "You don't do heavy lifting inside your Airflow worker." (12:47)
- Byron Allen: "The ability to do that, import them, build on top of them is just so nice." (27:23)

## Tools & references mentioned
- Servian
- Accelerate ML
- ML Test Score
- Airflow
- cron
- Kubernetes
- Kubernetes Pod Operator
- Kubernetes executor
- Kubeflow
- Polyaxon
- Redis
- RabbitMQ
- Postgres
- Snowflake
- Facebook Prophet
- MLflow
- Pandas
- Spark
- Informatica
- Elasticsearch
- Amazon S3
- Google Cloud Platform
- Stackdriver
- Celery executor
- Local executor
- Sequential executor
- Maxime Beauchemin
- The Rise of the Data Engineer
- Using Airflow with Kubernetes at Benevolent AI

## Who should watch
- You are deciding whether Airflow should coordinate model training, batch inference, or data preparation in an existing platform.
- Your team uses Kubernetes and needs to understand the extra operators and deployment choices required to run Airflow with it.
- You want practical guidance on DAG design, custom operators, logging, backfills, and idempotent workflows from people who have operated Airflow in production.

## Editor's note

Byron Allen says Airflow coordinates training and batch prediction while Kubernetes provides the runtime, so Airflow remains the coordinator rather than the complete ML platform. ZenML records each pipeline run's steps, inputs, outputs, and code version, giving the team a trace from a trained model back to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Airflow Sucks for MLOps](https://mlopstalks.com/talks/airflow-sucks-for-mlops) (Stephen Bailey, Whatnot, 1:05:53)
- [Why You Need More Than Airflow](https://mlopstalks.com/talks/why-you-need-more-than-airflow) (Ketan Umare, Union.ai, 1:11:12)
- [Making MLflow](https://mlopstalks.com/talks/making-mlflow) (Corey Zumar, Databricks, 59:11)
- [Packaging MLOps Tech Neatly for Engineers and Non-engineers](https://mlopstalks.com/talks/packaging-mlops-tech-neatly-for-engineers-and-non-engineers) (Jukka Remes, Haaga-Helia University of Applied Sciences, 8wave AI, 55:31)
- [MLOps vs ML Orchestration](https://mlopstalks.com/talks/mlops-vs-ml-orchestration) (Ketan Umare, Union.ai, 49:46)