Aurora uses Kubeflow Pipelines to automate the data, training, evaluation, export, and deployment stages in its autonomous-vehicle development loop.
2
Aurora separates Kubeflow orchestration from compute, allowing pipeline components to run on systems such as its batch-processing platform and AWS SageMaker.
3
The MLOps team increased adoption by building initial pipelines for teams, then moving toward consultation, reusable components, documentation, and small incremental pipeline builds.
Summary
Aurora's autonomous-vehicle workflow starts with vehicle logs, labeling, dataset creation, model training, evaluation, simulation, and deployment back to the vehicle. Before Kubeflow, many stages involved manual work that slowed experiments and made failures harder to trace. The team chose Kubeflow Pipelines after evaluating MLflow, Metaflow, and Airflow, partly because Aurora already had Kubernetes expertise. Its platform has separate build, orchestration, and compute layers. Pipelines provide automation, metadata, experiment tracking, a shared UI, and infrastructure abstraction. Maurizio Vitale describes a production deployment on AWS Kubernetes that keeps databases, artifacts, secrets, and authentication outside the cluster where possible. Aurora uses team workspaces rather than one namespace per user. Adoption required demos, white-glove implementation, internal documentation, and reusable components. The team tracks pipeline runs and pull-request verification as signs of usage.
Kubeflow sits inside an iterative autonomous-vehicle data loop
Vinay Anantharaman explains that Aurora begins with vehicles collecting sensor data, decisions, and actions in logs. After ingestion and labeling, Kubeflow creates datasets, trains models, runs evaluation, exports models to GPU queues, and connects the results to the wider perception and planning system. Developers then run simulation before deploying a model back to the vehicle. This loop repeats, so delays in any stage reduce the number of experiments and new capabilities the team can put on the road.
Automation and traceability were the reasons Aurora needed orchestration
Ankit Aggarwal describes manual workflow steps as the main source of delay. Engineers had to watch stages, handle failures, and retry work by hand. They also lacked a consistent way to track experiments, share them with colleagues, debug the full lifecycle, and reproduce datasets and models. Aurora wanted pipelines that could be automated, reused, extended, scaled across teams, and accessed through self-service. Security and limited infrastructure permissions were part of the design from the beginning.
Aurora separates pipeline orchestration from the systems that do the compute
Aurora's architecture has a build layer for Docker images and artifacts, a Kubeflow Pipelines layer for workflow orchestration, tracking, metadata, and visualizations, and a compute layer for intensive jobs. This separation lets the team send work to different systems, including homegrown infrastructure and AWS SageMaker. The arrangement keeps ML engineers focused on models while infrastructure engineers can run autonomy pipelines without needing deep knowledge of each model.
Reusable components connect model work to company systems
Aurora built shared Kubeflow components for batch processing, SageMaker training, TensorBoard, GitHub, and Slack. Its batch component launches dataset-generation and model-evaluation jobs, while a modified AWS component provides a simpler training interface with defaults. GitHub components comment on pull requests and support deployment checklists. Every pipeline has a Slack exit handler that reports failures or successful completion to the developer.
The developer workflow compiles pipelines as part of the monorepo build
Pipeline definitions live in Aurora's monorepo and use Bazel as the build system. A pipeline builder separates compile-time parameters from runtime parameters. Aurora compiles pipelines and components each time they run so Docker images and dependencies stay current. Developers can launch experiments from a CLI, use preconfigured Kubeflow commands in GitHub pull requests, or rely on daily and weekly CI jobs. A perception pipeline combines dataset generation, training, export, evaluation, and deployment in one UI.
Production Kubeflow keeps most state outside the Kubernetes cluster
Maurizio Vitale describes Aurora's move from local Kubernetes experiments to a multi-user AWS deployment. Production uses Kubeflow Pipelines and Jupyter notebooks on Amazon EKS, with artifacts stored in S3 instead of MinIO. Databases are managed outside the cluster so cluster maintenance does not threaten application state. Aurora also keeps secrets in Vault and integrates with its own authentication and authorization system. The goal is zero setup for users when they join a team.
Aurora uses team workspaces and external identity instead of one namespace per user
The deployment does not follow Kubeflow's default model of one namespace per user. Aurora creates one workspace per team and synchronizes corporate group information into Kubernetes roles and Istio rules. Users access Kubeflow through the company's Okta-based identity system, with proxy layers mapping identity headers to Kubeflow authentication. This reduces maintenance while allowing multiple members of a team to use shared pipelines and notebooks.
Adoption improved when the MLOps team built small examples and reusable starting points
Vinay says teams initially needed a clear explanation of Kubeflow's use cases and benefits before they would prioritize automation. Demos helped generate interest. Aurora first used a white-glove model, building pipelines for teams, and then shifted toward consultation. The team recommends starting with two pipeline steps, then adding more, because running an early version helps engineers turn judgment calls into explicit steps. Common components make each subsequent pipeline less work.
Pipeline usage gave Aurora a way to judge whether the platform was being adopted
Aurora started its Kubeflow effort in 2021 and tracked pipeline runs as more teams released pipelines in 2021 and 2022. The team saw hundreds of runs per month, which Vinay connected to ML engineers running experiments through Kubeflow while training jobs took multiple days. It also tracked pull requests that ran Kubeflow verification commands. These measures included activity from both ML and infrastructure engineers.
"The fastest way to actually get them to have a pipeline and to start using the infrastructure was white glove service."Vinay Anantharaman35:17
Who should watch
You are evaluating Kubeflow Pipelines for a large ML organization and want to see how it fits into an existing Kubernetes and cloud setup.
Your team needs to automate long-running training and evaluation workflows while keeping compute separate from orchestration.
You are responsible for platform adoption and need practical ideas for onboarding ML teams, building reusable components, and reducing pipeline setup work.
