Sadik Bakiu uses Kubeflow Pipelines to orchestrate Spark jobs running on Kubernetes.
2
A Kubeflow pipeline applies a Spark application manifest, polls the Spark job status, and continues only after the job completes.
3
The setup needs a Spark Operator, suitable Kubernetes permissions, and a dedicated service account for creating and managing Spark pods.
Summary
Sadik Bakiu explains why a Spark job needs orchestration when data processing becomes part of a repeated machine learning workflow. He focuses on preprocessing, where data is transformed and features are calculated, and shows how Spark can distribute that work across Kubernetes nodes. Spark Operator lets the cluster run Spark applications from Kubernetes manifests. Kubeflow Pipelines then wraps those operations into a multi-step workflow. The demo uses Minikube, Kubeflow, and the Spark Operator. One pipeline component applies a Spark application manifest, another repeatedly checks its status, and a final component prints a completion message. The example estimates pi. Sadik also covers practical limits. Spark may be excessive for data volumes around one or two gigabytes, network communication can slow a job, and permissions must be configured carefully. He explains that Kubeflow Pipelines uses Argo underneath and that the pipeline code can be stored and deployed through normal source control and CI/CD processes.
Repeated machine learning workflows need orchestration around distributed processing
Sadik Bakiu starts by separating a one-off Kubernetes job from a workflow that runs repeatedly. A machine learning workflow usually includes preprocessing, training, and serving. Preprocessing cleans data, applies transformations, and calculates features. Training produces one or more models, while serving makes those models available to users or business teams. New data feeds back into the process because the original training data can become old or irrelevant. The talk focuses on preprocessing. As input data grows, adding a larger machine is not always an option, so the workload must be distributed across more machines and coordinated around dependencies.
Distribution alone is insufficient when processing steps depend on one another
Spark can distribute computation, but Sadik explains that a processing workflow still needs to understand the order of its steps. Some operations need the output of an earlier step and must run sequentially. Other operations have no dependency and can run in parallel. Orchestration coordinates those relationships and helps use the available resources efficiently. Sadik also wants the processing workloads to run in the same infrastructure as related workloads. Keeping them separate can create data silos and require another infrastructure layer. Running them together can reduce the time and cost of provisioning that extra layer.
Spark Operator turns Spark applications into Kubernetes-managed workloads
Sadik introduces Spark as the distribution framework in the demo and Kubernetes as the container orchestration framework. Spark distributes work across nodes and aggregates results through the driver. Kubernetes distributes and controls workloads through its control plane. Spark jobs can be submitted with the spark-submit command, using Spark's native Kubernetes scheduler. The other option shown is Spark Operator, an open source tool that makes Spark jobs resemble other Kubernetes workloads. With the operator installed, the user defines a Spark application in a manifest and submits it with kubectl. Kubernetes then creates the driver and executor workloads.
Kubeflow Pipelines adds workflow structure around Kubernetes operations
Kubeflow is presented as a platform for deploying and working with machine learning workloads on Kubernetes. Sadik mentions training operators, model serving, Katib for hyperparameter tuning, and Kubeflow Pipelines, while limiting the demo to Pipelines. A pipeline defines components and their relationships. Each component is a container or manifest that runs in the cluster, often as a pod. Its component specification defines metadata, the interface, and the implementation. Sadik uses Kubernetes components so a pipeline can run commands such as kubectl apply and kubectl get against the cluster. This lets the pipeline coordinate Spark without treating Spark as a separate system.
The demo polls a Spark application until Kubernetes reports completion
The pipeline begins with an apply component that loads a Spark application manifest and runs kubectl apply. Once Kubernetes creates the Spark application object, that component finishes. A second Kubernetes get component receives the application name and periodically checks its status. It waits while the job is running and moves forward when the job reaches the completed state. Sadik adds a five-second sleep between checks to avoid sending excessive requests to the control plane. The status check is recursive, so the pipeline uses a graph component configuration that allows recursion. After the loop, a print operation reports that the job is complete.
The working example depends on permissions and cache settings
For the demo, Sadik uses Minikube, installs Kubeflow, and installs Spark Operator with Helm. He creates a service account, a role with permissions to create and delete pods, and a binding between them. The pipeline is run with that service account because Spark needs permission to create its applications and pods. He also disables caching for the apply and status operations. Each pipeline run should submit and execute the Spark job rather than reuse a cached result. When he runs the example, the pipeline starts the Spark driver and executors, checks the status, and prints completion after the Spark job finishes.
Spark is a poor fit for small data and can be slowed by cluster overhead
In response to Demetrios Brinkmann, Sadik says Spark is a trade-off. For production processing involving gigabytes or terabytes of data, he would generally choose Spark. If the data is around one or two gigabytes or less, Spark may be overkill, and pandas or another framework may be enough. He also warns that a slow Spark job may be spending time on communication between pods rather than on its own computation. Teams should watch network overhead and avoid changing the grouping of data frames during execution without understanding the effect. Permissions are another concern, since the Spark Operator needs authority to create pods.
Kubeflow Pipelines can fit normal source control and CI/CD practices
Sadik says Kubeflow pipeline definitions are Python files, so teams can manage them like other Python applications. They can store the files in their preferred source control system and deploy them through their existing process. Kubeflow also provides an API for updating and deleting pipelines. He explains that Kubeflow Pipelines uses Argo underneath, adding machine learning context around the orchestration. Sadik briefly mentions Tekton as another possible option, while noting that it is not coupled to Argo in the same way. His answer keeps the integration practical: the pipeline code can be versioned, reviewed, and delivered as part of a CI/CD process.
