ZenML moved from a client-driven design to a client-server architecture so teams could centralize permissions, databases, collaboration, and pipeline state.
2
ZenML separates pipeline code from stack configuration, allowing the same pipeline to run locally, on Vertex AI, or on a Kubernetes and Kubeflow setup without changing the pipeline code.
3
A GitHub Actions workflow can connect to a ZenML server, select a staging or production stack, run the pipeline, and deploy a model after evaluation.
Summary
Hamza Tahir explains why ZenML had to rebuild its architecture after early shortcuts created too much technical debt. The 0.20 release moved ZenML from a client-driven model to a client-server design with a central API, database, dashboard, authentication, and permissions. The main abstraction remains a pipeline written with Python functions and decorators, while a stack defines infrastructure such as the orchestrator, artifact store, experiment tracker, container registry, and model deployer. This separation lets the same pipeline move from local development to staging and production. In the demonstration, a local Iris classifier uses caching and artifact lineage, then runs on Vertex AI for staging and Kubeflow on AWS for production. GitHub Actions selects the appropriate stack, runs the pipeline, records experiments in MLflow, and updates a KServe deployment after evaluation. Hamza also describes the design-document and squad process ZenML used to reach consensus during the rewrite.
ZenML rebuilt its architecture after technical debt slowed development
ZenML initially took architectural shortcuts to get feedback quickly. Hamza says the resulting technical debt eventually made the team too slow, so they paused feature work for roughly two months and redesigned the framework. Engineers created milestones, wrote tests, mapped out the roadmap, and then merged the rewritten work. The final merge contained around 30,000 to 40,000 lines of code. The team had to keep the existing product usable while changing its internals, which made the rewrite difficult for an open source project with active users.
Pipelines provide a common interface across MLOps roles and tools
Hamza describes ZenML as an extensible open source framework for creating unified MLOps pipelines. The framework gives people with different roles, backgrounds, and responsibilities a shared interface and a common record of what happened. A pipeline is made from Python functions decorated as steps. The code defines the workflow, while the stack defines where and how it runs. This lets users keep pipeline logic separate from infrastructure choices and change the underlying tooling without rewriting the pipeline.
Stacks keep infrastructure configuration outside the pipeline code
A ZenML stack combines components such as an orchestrator, artifact store, experiment tracker, container registry, and model deployer. The artifact store controls where returned data and models are persisted, such as a local path, S3, or a Google Cloud bucket. The orchestrator controls where the steps run, which could be a local Python environment, Kubernetes, Kubeflow, Airflow, or Vertex AI. Pipeline code can therefore move between environments while the stack supplies the infrastructure-specific settings.
The client-server design centralizes state and access control
The 0.20 release changed ZenML from a client-driven architecture to a client-server architecture. In the earlier design, clients had local state and had to keep it synchronized with an external database. That created technical complexity and gave individual clients permissions they might not need for production workloads. The new design puts a central server between clients and the database. The server can run in a controlled environment behind a firewall, while clients use a standard API. The design also enabled ZenML's new dashboard.
ZenML adds a dashboard while keeping local use simple
ZenML's new dashboard gives users a visual view of pipelines, runs, stacks, and stack components. Hamza says the dashboard is bundled into the Python package, so users can start a local server and web application from the package itself. The team also built ZenML deploy to ask a few questions and deploy a server to the cloud. This keeps the entry point simple for users who do not already know Helm, Terraform, or GitOps, while leaving more advanced deployment options available.
Caching and lineage make repeated pipeline runs easier to inspect
In the notebook demonstration, Hamza runs a small scikit-learn classifier on the Iris dataset and views the run in the local dashboard. When he runs the same pipeline again, ZenML detects that the relevant steps have already run on the selected stack and reuses the cached results. The dashboard shows the pipeline graph, materialized artifacts, and the location of the stored outputs. Hamza explains that this becomes more useful with larger datasets, because preprocessing can be reused while later steps continue from the existing artifacts.
The same pipeline can move from local development to cloud staging and production
Hamza presents a workflow with separate local, staging, and production stacks. The local version reads local data. The staging version uses cloud data and runs on Vertex AI, while the production version uses an AWS stack with Kubeflow, an S3 artifact store, MLflow, and KServe. The pipeline functions stay the same. Only the selected stack and environment-specific data loader change. This lets teams review changes in staging before running the production pipeline and deploying a model.
GitHub Actions can connect code changes to training and deployment
The GitHub workflow in the demonstration connects to the central ZenML server, selects the Vertex AI staging stack, and runs the staging pipeline. After the pull request is merged, another workflow selects the production Kubeflow stack and runs the production pipeline. ZenML can build and push the required Docker image using the container registry defined in the stack. The production pipeline reads production data, records the experiment, evaluates the model, and updates an existing KServe deployment when the deployment condition is met.
ZenML used asynchronous design proposals to coordinate the rewrite
When Ben Epstein asks how the team reached agreement on the new architecture, Hamza describes a squad-based process. The team split into groups, created detailed design documents, and returned with proposals. Engineers commented on the documents asynchronously before holding meetings to resolve remaining issues. Hamza says this process suited a distributed team spread across Malaysia, India, and Munich. Squad leaders made many implementation decisions, and he trusted the engineers rather than trying to approve every detail himself.
"The technical debt bubble got us to a point where we were just too slow and we had to really stop, pause, take a two-month sort of break from features and re-architect the entire ZenML architecture."Hamza Tahir01:22
Who should watch
You are building machine learning pipelines that need to move from notebooks into shared staging and production environments.
Your team wants to change orchestrators, artifact stores, or experiment trackers without rewriting pipeline logic.
You are maintaining an open source framework and need a concrete example of planning a large architecture rewrite across a distributed team.
