ZenML gives data scientists a way to connect pipeline steps, artifact storage, metadata tracking, and infrastructure without rewriting their code for each environment.
2
The workshop builds an NBA pipeline that detects data drift with Evidently, tracks training runs with MLflow, sends alerts to Discord, and runs on Kubeflow Pipelines.
3
ZenML uses typed steps, persisted artifacts, metadata, and input-based caching to support reproducibility, while allowing custom materializers and integrations.
Summary
Hamza Tahir introduces ZenML through a live NBA example. The pipeline compares basketball data before and after Stephen Curry's change in three-point strategy, then uses Evidently to detect drift. The workshop shows how ZenML turns Python functions into typed pipeline steps, persists their outputs in an artifact store, and records metadata in a metadata store. It then adds MLflow for experiment tracking, Discord notifications for drift alerts, and Kubeflow Pipelines with a container registry for scheduled execution. A separate inference pipeline selects the best model from previous training runs and uses it to make predictions. Tahir argues that ZenML should connect existing tools rather than replace them. The same pipeline code can move from a local stack to a Kubeflow-based stack by changing the configured infrastructure components. He also explains how developers can add integrations, custom materializers, and their own orchestrators.
ZenML grew from the difficulty of putting production ML tools together
Hamza Tahir describes building predictive-maintenance systems around 2017, using sensor data such as pressure and temperature from trucks, buses, and trains. His team needed pipelines that were reproducible and reliable, but there was no easy way to move a data scientist's work into production with suitable abstractions. They solved parts of the problem in-house, then combined those lessons into ZenML around 2020. The goal was to let data scientists keep ownership of their models while connecting the tools used during experimentation with the tools used in production.
ZenML separates pipeline code from the infrastructure that runs it
Tahir presents ZenML as a framework for plugging different tools into a common pipeline structure. A data scientist can move from a notebook or local training setup toward a production pipeline without changing the basic pipeline code. The framework groups infrastructure components into stacks, such as an orchestrator, artifact store, metadata store, and container registry. This lets a project use local components during development and replace them with Kubeflow Pipelines or cloud services later. Tahir compares the intended experience to a web framework with useful abstractions and built-in components.
A ZenML step is a typed Python function whose outputs become tracked artifacts
The workshop defines a step by decorating a Python function with ZenML's step decorator. The function must use type annotations, which lets ZenML check that the data passed between steps has the expected type. A step can return one or several values, and those values become artifacts. ZenML persists artifacts in an artifact store and tracks them through a metadata store instead of passing every result only in memory. The framework includes materializers for common types such as pandas and NumPy data, while custom materializers can define how special objects are stored and loaded.
The NBA example uses historical changes in three-point shooting to demonstrate drift
The first pipeline uses NBA data to examine whether Stephen Curry's rise in long-distance shooting changed the distribution of three-point attempts. Tahir splits the data around February 27, 2016, then compares a reference dataset with a later dataset. The pipeline has an importer, a data-splitting step, and a drift-detection step. Evidently calculates the drift report, and ZenML exposes the result through a notebook API and visualizer. In the displayed result, drift is detected for all of the features being examined, which supports the example's claim that the game changed after Curry's strategy became common.
ZenML caching reuses step outputs when the inputs and code have not changed
Tahir explains that ZenML persists each step's output and records the inputs, parameters, upstream artifacts, and code hash. When those parts of a step's input signature remain the same, ZenML assumes that the output will also remain the same and reuses the stored artifact. This explains why a second pipeline run is faster. A step or an entire pipeline can disable caching when it depends on external state, such as a changing API. That setting forces the step to execute again even when its recorded inputs appear unchanged.
ZenML connects drift detection, experiment tracking, and scheduled orchestration
The second pipeline models a more regular production workflow. It imports data, performs feature work and encoding, trains a model, evaluates it, and checks whether recent data has drifted from the previous reference period. ZenML integrates with Evidently for drift reports and with MLflow for experiment tracking, including parameters, metrics, and models. The example also sends drift status to Discord. Tahir's point is that ZenML links these tools through pipeline and step runs, so training activity and operational signals can be connected without replacing MLflow or Evidently.
Changing the active stack moves the same pipeline onto Kubeflow Pipelines
The workshop replaces the local orchestrator with Kubeflow Pipelines and adds a container registry. ZenML builds a Docker image around the pipeline code, pushes it to the registry, and runs the pipeline through Kubeflow. The example uses a local Kubernetes setup for the demonstration, but Tahir describes the same arrangement with cloud infrastructure, shared artifact storage, metadata storage, and a remote container registry. A scheduled run can execute every 30 seconds in the demo, while a real system might run weekly or monthly. The stack configuration determines the infrastructure while the pipeline code remains largely the same.
Inference can select the best recorded model and preserve the path back to training
The third pipeline handles inference separately from training. It imports the latest data, applies the same preprocessing, selects a model from previous training runs, and generates predictions. The model-picker step examines recorded training runs, reads the test score, and returns the model and run ID associated with the best score. Tahir notes that this logic could also query MLflow. Because ZenML tracks pipeline runs, steps, artifacts, and metadata, a prediction can be traced back to the model that produced it and then to the training pipeline and its component steps.
"The goal of ZenML is to be able to swap out these pieces of infrastructure for you and provide the standardization layer which links everything together."Hamza Tahir55:19
Who should watch
You are building ML pipelines in notebooks and need a path toward scheduled, containerized execution.
Your team already uses tools such as MLflow or Evidently and wants to connect them without replacing them.
You want to understand how artifact persistence, metadata, caching, and model selection fit into an MLOps workflow.
