Kubeflow brings notebook development, training, pipelines, tuning, serving, metadata, and monitoring together for machine learning on Kubernetes.
2
Kale and Arrikto storage let data scientists turn notebook code, data, and metadata into reproducible Kubeflow Pipelines without manually building containers or writing pipeline DSL code.
3
Kubeflow 1.0 focused on product hardening, component versioning, documentation, and workflows that support portable, repeatable machine learning across laptops, clouds, and on-premises systems.
Summary
Josh Bottum presents Kubeflow 1.0 as a composable set of tools for developing, training, deploying, and managing machine learning on Kubernetes. He argues that teams spend much of their time on work around the model, including data preparation, resource management, tuning, serving, monitoring, and reproducibility. Kubeflow connects notebooks, training operators, Pipelines, Katib, TensorBoard, metadata, and KFServing so these tasks can be handled in one Kubernetes-based environment. The demonstration focuses on the notebooks-to-pipelines workflow. With Kale, a data scientist labels notebook cells, snapshots the code and data, and creates a pipeline from the notebook. Arrikto's persistent-volume and snapshot capabilities provide storage for workspaces, datasets, and reproducible pipeline steps. Bottum also explains Kubeflow's release process, component maturity levels, and its relationship with tools such as Airflow. He is direct that Kubeflow still requires community involvement and that some capabilities, including serving models directly from the dashboard, were still on the roadmap.
Kubeflow covers the work around a model as well as the model code
Bottum says machine learning projects often fail to reach production because the surrounding work takes more effort than writing the model. That work includes collecting and verifying data, extracting features, managing machines and GPUs, tuning hyperparameters, reusing previous work, serving models, and monitoring production. A team may write a model in two weeks and then spend six months deploying it. Kubeflow addresses these parts through notebook tools, training operators, workflow management, serving infrastructure, and monitoring integrations. The goal is to let a model move from a laptop or IDE into training and production without rebuilding the operational process each time.
Kubeflow is designed to move workloads across Kubernetes environments
Kubeflow's mission is to make it easier to develop, deploy, and manage portable, distributed, and scalable machine learning systems on Kubernetes. Bottum describes portability as moving data, models, and metadata from a laptop to a public cloud or an on-premises system. Kubernetes provides declarative operations for placing and autoscaling workloads, while Kubeflow's components handle machine learning tasks on top of that infrastructure. The same architecture is intended to support a single user or a much larger environment with different compute, storage, and memory resources.
Kubeflow 1.0 added process and product hardening around its components
Bottum describes Kubeflow's progression from individual applications such as Jupyter notebooks, TFJob, and TensorFlow Serving to connected workflows with Pipelines, Katib, KFServing, Fairing, and metadata. Kubeflow 1.0 continued the productization work. The community published a roadmap, defined a versioning policy, and assessed components as stable, beta, or alpha. The application requirements template covers configuration, deployment, custom resources, logging, monitoring, container images, CI/CD, documentation, testing, ownership, and adoption. Bottum presents this process as part of making the project usable, since different components mature at different rates.
Kubeflow uses composable components instead of forcing one workflow tool
Bottum describes Kubeflow as a set of software components that can be combined according to a team's needs. Notebooks provide an interactive machine learning development environment. Training operators place distributed workloads on suitable infrastructure. Fairing helps with containerization, training, and deployment. Pipelines make workflows repeatable, while Katib tunes hyperparameters and TensorBoard helps inspect models and datasets. Kubeflow can also work with other workflow runners. Bottum gives Airflow as an example, where a team might keep its data pipeline in Airflow and use Kubeflow for machine learning pipeline work.
Persistent volumes and snapshots give Kubeflow shared, repeatable data workflows
Bottum explains that Kubeflow uses Kubernetes persistent volumes for data management between systems and users. These volumes support sharing, versioning, and reproducibility. Arrikto adds a Kubernetes Container Storage Interface storage class and supports operations such as dynamic volume provisioning. In the notebook interface, users can choose a workspace volume for code and packages and a persistent data volume for datasets. Arrikto's snapshots capture the notebook's code, data, and metadata. Bottum says local-disk architectures can be faster and less expensive than remote storage, while the same storage approach can also connect to remote arrays or cloud services.
Kale turns a notebook into a Kubeflow Pipeline with less manual packaging
The notebooks-to-pipelines critical user journey is the main demonstration. A data scientist labels notebook cells with Kale, selecting imports, data loading, processing, feature engineering, and model steps while skipping exploratory cells that should not run in the pipeline. Kale removes the need to manually build Docker containers, write pipeline DSL code, compile it, and upload it. It snapshots the notebook environment, then builds and runs a containerized pipeline in Kubeflow Pipelines. The resulting graph shows how data and artifacts move between steps. Bottum says this changes the process from writing code, containers, and DSL by hand to tagging notebook code and creating the pipeline with a button.
Snapshots let researchers branch from an earlier pipeline step
In the Titanic example, the initial models all reached perfect accuracy because the target column, survived, was still included in the feature data. Bottum opens the artifacts from a random forest step and uses the Arrikto snapshot to create another notebook with the saved code and data. The snapshot is immutable, so the earlier state remains reproducible, while the new notebook can change the workflow. After dropping the survived column, the random forest result falls to 92 percent. Bottum then reruns the updated notebook through Kale and creates another pipeline iteration. This demonstrates how a researcher can inspect a result, recover the exact working state, correct the code, and repeat the experiment.
Kubeflow still depends on an open community and continued product work
Bottum attributes Kubeflow's growth to a welcoming community that includes contributors from Google, IBM, Red Hat, Microsoft, Seldon, GoJek, and other organizations. He says a recurring portion of new users are new to the community, while users and vendors contribute to different components. He is also clear that Kubeflow is not a fully managed vendor product and requires users to do some of their own care and feeding. During the questions, he says model serving from the dashboard was not yet available, although the community was considering a workflow from notebooks through tuning and deployment. Users can join the community, follow the roadmap, and contribute use cases or issues.
"Kale leverages Arrikto to automatically take snapshots of your complete notebook, both the code and the data and the metadata, and then it builds and runs containerized pipelines."Josh Bottum31:22
Who should watch
You are building machine learning models in notebooks and need a repeatable path into training and deployment on Kubernetes.
Your team spends more time packaging models, managing data, and coordinating infrastructure than writing model code.
You are evaluating Kubeflow 1.0 and want to understand how persistent volumes, snapshots, Kale, and Kubeflow Pipelines fit together.
