Stephen Bailey argues that Airflow's widespread use has shaped how data teams think about orchestration, even when its task-and-schedule model does not fit machine learning workflows.
2
Whatnot separates its data platform, analytics, and machine learning responsibilities, then centralizes scheduling so DBT, SageMaker, ingestion, and training jobs can depend on one another.
3
Stephen recommends moving uncomfortably fast while distinguishing reversible decisions from expensive data-modeling decisions that become part of the infrastructure.
Summary
Stephen Bailey describes his work on Whatnot's data platform and explains why he thinks Airflow limits conversations about orchestration. Whatnot split its data platform, analytics, and machine learning teams early, giving each a clearer responsibility. The platform team manages orchestration and data movement, while supporting analytics, recommendation systems, trust and safety, and product use cases. Stephen wants one central scheduler rather than separate schedules inside DBT Cloud, SageMaker, ingestion tools, and other systems. Whatnot uses Dagster, including its assets framework, to connect jobs to upstream data dependencies. Stephen prefers Dagster's focus on the data engineer, but his larger point is about the model of orchestration. Airflow encourages teams to think in independent scheduled DAGs. DBT encourages thinking about a graph of data assets. Machine learning adds another model centred on the lifecycle of a trained model. Stephen also explains how data classification, modular jobs, and explicit ownership help teams share platforms without losing track of dependencies.
Whatnot gives its data platform a narrow enabling role
Stephen describes Whatnot as a live-stream retail company with separate analytics, data platform, and machine learning teams. The data platform team owns orchestration and the pipelines that move data from the main application systems into operational machine learning, analytics, and other uses. The team supports domains such as trust and safety and product experience, while mainly enabling other teams to build advanced applications. Stephen says the separation let each group develop a more specialised mindset instead of one team trying to do everything poorly. He also values working alongside engineers who have already built systems at scale.
Business value and platform design need separate attention
Stephen describes a tension between solving an immediate business problem and building a platform that works systematically over time. At Whatnot, analytics and data platform work were separated from the beginning. Analytics focused on board meetings, definitions, and answering questions, while the platform and machine learning side needed rigorous systems and pipelines defined as code. The team also created a data quarantine, separated from lines of business, where it could consider orchestration, reuse, and developer experience. This structure lets product teams move toward value quickly while giving platform engineers a place to make broader improvements.
Whatnot started with metrics, recommendations, and trust and safety
The first data priorities were tied to Whatnot's customer journey. The team needed reliable, consistent metrics for orders, shows watched, time spent in shows, and product engagement. Once those base metrics were available, recommendation became a major machine learning use case, with the aim of driving an algorithmic feed and further experimentation. Trust and safety was another early priority because live streams create ways for users to abuse or damage the experience. That work required both faster access to data and shared definitions for problematic activity, so the information could reach agents. Stephen stresses that analytics, machine learning, and operational workflows have different stakeholders and different data flows.
Real-time recommendations depend on precomputed data and fast serving
Whatnot's recommendation and other product-serving systems need real-time responses, low latency, and the ability to handle high workloads. Stephen says the team uses Rockset as a real-time analytics layer. Offline processing produces data that is dumped into Rockset, where fast lookups and SQL queries make the results available to other services through an API. The main application can query that API and filter the results for particular users. Many recommendations are precomputed and placed there before the product requests them. Stephen is less involved in the internal training details, which he identifies as a boundary between data platform engineering and machine learning engineering.
Central scheduling keeps separate tools from becoming separate systems
Whatnot moved from Spell into the SageMaker ecosystem for machine learning workflows. The data platform team provides reusable patterns where data scientists supply a Docker container, shell script, configuration, and parameters for a SageMaker training job. They can copy the setup into the Dagster orchestrator and run it on a schedule. Stephen's broader orchestration rule is that scheduling should be centrally managed. DBT Cloud, SageMaker, ingestion pipelines, and other tools should not each run their own independent schedules. A central scheduler can connect dependencies, such as making a SageMaker job wait for a DBT job, and avoid downstream jobs running on stale or incomplete data.
Airflow's task model can narrow how teams think about data
Stephen says he wrote about Airflow after seeing data products, trust and safety jobs, training jobs, DBT pipelines, and other systems become difficult to coordinate. Airflow was the obvious place to address the problem because it is connected to so many parts of the infrastructure, but he felt that orchestration tools were not giving teams a complete view of what existed or helping them manage the parts together. In his view, Airflow's model of independent scheduled DAGs encourages engineers to think in separate processes that run on schedules and can be backfilled. DBT presents a different model, where engineers think about a graph of data assets and how data moves between them.
Machine learning needs orchestration centred on models
Stephen says machine learning workflows have an ontology that differs from traditional data processing. The important object is often the model, its retraining history, and what happens after a retraining, rather than a series of dated processing runs. Users want to inspect the model and its history, including whether it has been deployed. Because existing orchestration systems often focus on scheduled data jobs, teams may build another orchestrator for machine learning. Stephen would prefer fewer systems, but he says the distinction between data processing and model lifecycle management makes that difficult. He also notes that the machine learning world has received enough venture funding that consolidation is unlikely soon.
Modularity and data classification let teams share a platform
Whatnot has several repositories that independently send jobs to a central Dagster scheduler. Separate Docker containers and package requirements allow the platform to support very different workflows, including data science and machine learning jobs. Stephen says the harder problem is helping teams understand which data products are dependable and how their work connects to other teams. Whatnot divides its DBT project into raw data marts, core Kimball-modeled marts, product-specific marts, and scratch spaces. Users can choose raw data, dimensionally modeled data with team guarantees, or more highly vetted data published for machine learning. Dagster's assets framework then lets a machine learning job declare the DBT table it depends on, making the relationship visible for triggering, troubleshooting, and support.
Move quickly while treating irreversible decisions carefully
Whatnot's company philosophy is to move uncomfortably fast and avoid overthinking the first version. Stephen prefers finding an 80 percent solution, putting it in front of internal stakeholders, and iterating. He makes an exception for data modeling because published models become part of the infrastructure and are expensive to replace once other work depends on them. The team therefore separates type-one decisions, which can be reversed, from type-two decisions, which require more care. Stephen also describes an 'own everything and nothing' approach. The platform team needs visibility into events, schemas, validation, trust and safety data, and downstream uses across the company, even when it does not directly manage every component.
"We try to minimize, figure out the 80 percent best solution and then go with it as fast as possible."Stephen Bailey1:01:15
Who should watch
You are deciding whether Airflow should remain the shared scheduler for analytics, data pipelines, and machine learning jobs.
Your teams use several tools with their own schedules and need dependencies, ownership, and operational visibility in one place.
You are building a data platform and want practical guidance on separating platform work from analytics while keeping model and data dependencies explicit.
