A general workflow tool such as Prefect, combined with a compute service such as Modal, can cover many machine learning workflows with less framework-specific code.
2
ZenML and Metaflow target data scientists and platform engineers, while tools such as Airflow, Dagster, and Prefect provide broader workflow orchestration.
3
The right choice depends on the team and its work. Large teams may value the uniformity of an opinionated platform, while small AI startups may prefer simple compute and orchestration tools.
Summary
The discussion compares ZenML and Metaflow with general workflow tools such as Prefect and compute platforms such as Modal. Ben argues that many teams can use ordinary Python, a generic orchestrator, and a compute service while keeping checkpointing, experiment tracking, and deployment relatively simple. Eric is more cautious because he has seen data-science-specific DAG tools solve problems around heterogeneous compute and reproducibility. The conversation then moves to team structure. Data scientists often own business problems, while engineers build reusable systems, so platform teams can provide a self-service layer instead of passing models over a handoff. ZenML and Metaflow fit inside that kind of platform, but they also introduce framework conventions and some lock-in. The speakers agree that there is no universal answer. Enterprise teams may want the consistency and collaboration features of an opinionated DAG tool. Small AI startups may gain more from a simple compute service. They also argue that data transformations remain a separate and difficult part of the stack.
Generic orchestration can cover many machine learning workflows
Ben argues that a broad tool such as Prefect can schedule workflows, respond to events, run ordinary Python, and handle work outside machine learning. He uses it for automated emails as well as model training, while tracking experiments in Weights & Biases and checkpointing models. With compute supplied by Modal or a similar service, a team may need fewer specialized tools. The tradeoff is that specialized systems can provide features such as model checkpointing and reproducibility directly, so the simpler approach still requires careful implementation.
The handoff between data science and engineering creates operational problems
Ben describes an earlier project where data scientists trained a random forest model in scikit-learn, saved it as a pickle, and handed it to data engineers to deploy, monitor, and update. The arrangement created repeated questions about performance, dependencies, broken outputs, and ownership. A model's performance might mean speed, resource use, or prediction quality, and those concerns cross the boundary between the two teams. His conclusion is that handing a product to a group that did not create it makes maintenance harder because the new owners lack context.
Platform teams can give data scientists a self-service path without requiring unicorns
The speakers distinguish the vertical problems that data scientists care about, such as recommendations and customer churn, from the reusable systems that engineers prefer to build. A platform team can create the infrastructure, workflows, and patterns, while a consuming team uses them without a formal handoff. That lets the consuming team own its work from modeling through monitoring and updates. The approach does not require every data scientist to understand every cloud, infrastructure, orchestration, and observability tool. The platform provides the common parts instead.
ZenML and Metaflow target both data scientists and platform engineers
ZenML and Metaflow are discussed as tools that primarily attract data scientists but are also implemented by platform engineers. Ben says Hamza, ZenML's CTO, described platform engineers as first-class users of ZenML. The tool is intended to accommodate engineers with strong infrastructure preferences rather than forcing one fixed infrastructure choice. This makes the platform persona part of the product's design, alongside the data scientist who uses the resulting workflows.
An MLOps platform is made from separate jobs to be done
Ben presents ZenML's stack concept as a way to break an ML platform into separate jobs. The stack can include code storage, experiment results, feature storage, offline compute, online inference compute, and other services. In principle, an orchestrator or compute component can be replaced without rebuilding everything else. He says this matters because the MLOps market is young and tools can change quickly. Eric challenges the practical side of that idea, arguing that replacing an entire opinionated system such as ZenML with Metaflow would still be a large project.
Simple components can preserve useful reproducibility
Ben argues that teams can reproduce an experiment with a relatively small set of artifacts: the input data, the code that trained the data, and the hyperparameters. He also mentions storing the trained model, using an MLflow model in a Docker image to manage dependencies, and tracking experiments with Weights & Biases or MLflow. For production, he describes a conventional FastAPI service with logging, alerts, resource thresholds, and prediction data written back to a database. Evidently or Grafana can then be used for drift checks or dashboards.
Specialized DAG tools simplify data-science workflows at the cost of convention
Eric compares the general DAG tools Airflow, Dagster, and Prefect with Metaflow, ZenML, and Flyte. The latter group was built with data scientists in mind and makes heterogeneous compute, such as using different resources at different DAG steps, easier. Metaflow requires flows, classes, step decorators, and explicit transitions. ZenML uses step and pipeline decorators. These conventions can reduce a large amount of configuration to a small amount of code, but they also mean that users must learn the framework and accept its abstractions.
Team size and type of work should determine the amount of structure
Ben says platforms are useful when a team has many units of work and wants consistent standards. That fits teams doing repeated regression and classification projects for business stakeholders. It differs from a small company building one compute-heavy product, such as live translation of basketball commentary, where a full DAG tool may add little value. Demetrios adds that larger teams may need uniformity and manageable change, while a six-person team starting from scratch can adopt a preferred design more easily than a larger team with established practices.
Data transformation can matter more than the model workflow
Ben argues that uniformity may be more valuable in the data platform than in the machine learning layer. He praises SQLMesh for checking downstream dependencies, detecting breaking changes, requiring GitHub-based changes, and providing a controlled backfill process. The speakers question whether ZenML can cover the path from raw data to a trustworthy training set. Reproducing the final model does not solve a problem if the upstream data transformations are poorly understood. This is why Ben is interested in making the data transformation layer more consistent.
"The hardest thing was just getting your GPU CUDA kernels to match the drivers that were installed when you spun up that EC2 instance."Ben37:49
Who should watch
You are choosing between a general-purpose orchestrator and a machine-learning-specific workflow tool for a small or medium-sized team.
Your data scientists and engineers are passing models across a team boundary and you need to decide whether a platform layer would reduce that friction.
You are designing an ML platform and need a practical discussion of framework lock-in, heterogeneous compute, reproducibility, and data transformation.
