# MLOps at Volvo Cars

Leonard Aukea, Volvo Cars | MLOps Meetup | Episode 84 | 57:31
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=VzgomadGo1g
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mlops-at-volvo-cars
Published: 2021-11-08
Tags: monitoring, orchestration, platform-teams, testing

## TL;DR
- Volvo Cars uses cloud-native infrastructure, Kubernetes clusters, Kubeflow, Tecton, GitHub, and internal templates to reduce friction for machine learning teams.
- The platform connects Git workflows with reproducible pipelines, model and data versioning, testing, deployment, and team access controls.
- Leonard Aukea argues that MLOps depends on engineering practices and culture, including software testing, behavioral regression tests, code review, and stronger software skills among data scientists.

## Summary
Leonard Aukea presents Volvo Cars' approach to building a machine learning platform for enterprise use. The company runs dedicated machine learning clusters on Kubernetes and uses Kubeflow for workflows, distributed training, and team isolation. Tecton connects Git repositories with pipeline automation, image building, testing, and deployment patterns. Internal project templates set up access, sandboxes, naming conventions, and other defaults so teams can start with less platform work. Volvo Cars uses Harbor and an internal CLI tool to store model artifacts, while LakeFS is being explored for data versioning. Leonard also describes gaps in monitoring, feature versioning, and machine learning testing. He treats testing as a broader practice than model evaluation, with explicit checks for failure modes and behavioral regressions. The talk also covers team culture, software engineering expectations for data scientists, cloud agnosticism, distributed ownership, and the possibility of federated learning for connected cars.

## Key ideas
### Volvo Cars starts with a cloud-native platform and dedicated machine learning clusters
[03:05](https://www.youtube.com/watch?v=VzgomadGo1g&t=185s)
Volvo Cars chose cloud-native infrastructure because it already used container orchestration for its connected car cloud and other services. Machine learning workloads run on separate clusters because their needs differ from those of general web applications. Kubeflow is the central component because it provides an open-source unified interface, workflow abstractions, pipelines, and operators for distributed training. Leonard Aukea says Kubeflow lacked the CI/CD layer Volvo Cars needed, so the team built tasks and pipelines to connect Git repositories with the Kubeflow Pipelines API.

### Project templates automate the setup that every machine learning team needs
[09:39](https://www.youtube.com/watch?v=VzgomadGo1g&t=579s)
Volvo Cars uses a GitHub repository template as the starting point for machine learning projects. Teams add a bot user, which enables the CI automation, and define access in a profiles.yml file. That file controls who can access a project, while a Kubeflow profile creates an isolated namespace. The template also prepares a LakeFS sandbox, credentials, defaults, linting, code formatting, and other project settings. Leonard describes this onboarding as a way to remove boilerplate and help teams use shared patterns instead of rebuilding the same setup.

### The Git workflow versions pipelines and supports controlled experiments
[07:47](https://www.youtube.com/watch?v=VzgomadGo1g&t=467s)
Each GitHub branch gets a corresponding naming convention in the pipeline system, so teams can track which pipeline version they are using. Automation builds and tags images for components, runs standard software tests and quality checks, and enables orchestrated experiments with comparisons. Leonard uses shadow deployment as an example: a new model can run in parallel on the same live data as the customer-facing model while remaining hidden from customers until its behavior has been checked. Pipeline runs stay on demand because training a large model on every commit would not make sense.

### Volvo Cars reuses infrastructure for model and data versioning
[15:20](https://www.youtube.com/watch?v=VzgomadGo1g&t=920s)
The team uses an internal CLI tool called RNB to store serialized models in Harbor, its image registry. Models carry metadata and tags, and can be referenced from an inference service manifest using KServe. Volvo Cars previously experimented with DVC but found that it did not scale for them. The team is exploring LakeFS as a Git-like system for data. Because LakeFS did not provide the exact repository pointer they wanted in the Git repository, they implemented that connection in their internal MLOps toolkit.

### Monitoring needs interfaces that machine learning teams can own
[34:33](https://www.youtube.com/watch?v=VzgomadGo1g&t=2073s)
Volvo Cars uses Thanos, Prometheus, and Grafana for general cloud-native monitoring, but Leonard says these tools do not cover every machine learning need. Adding a custom metric currently requires defining it, configuring Prometheus to scrape it, and creating a Grafana chart. He considers that process awkward for data scientists. The team wants model developers to own the metrics, alert thresholds, and monitoring rules for their systems because they understand the intended behavior best. Volvo Cars has discussed these needs with Arize, while improved monitoring remains on its roadmap.

### Machine learning testing checks behavior beyond evaluation metrics
[27:29](https://www.youtube.com/watch?v=VzgomadGo1g&t=1649s)
Leonard separates model evaluation from model testing. Evaluation summarizes performance on validation or test data with metrics, plots, operational statistics, and examples. Testing adds explicit checks for behaviors the model is expected to follow. A failure mode is a scenario where the model fails, while a behavioral regression is a performance decrease on specific examples. When a failure mode appears, the team should add a regression test and require future model versions to pass it. Leonard also proposes a machine learning red team that stress-tests systems before release, monitors them in operation, and tracks failure modes.

### The platform depends on software engineering habits and shared culture
[34:02](https://www.youtube.com/watch?v=VzgomadGo1g&t=2042s)
Leonard says tools alone are insufficient. Teams need shared guidance, internal documentation, design patterns, and a culture of review. Volvo Cars has found that data scientists often need more practice with Git, collaboration on larger codebases, CI/CD, containers, testing, and software design. The company promotes a 'data scientist plus' profile, while avoiding a requirement that data scientists become DevOps engineers. The goal is to make work maintainable enough for handover between data scientists and machine learning engineers. Peer review matters because machine learning work involves complex data and many ways for a system to go wrong.

### Cloud agnosticism and distributed ownership shape the architecture
[41:48](https://www.youtube.com/watch?v=VzgomadGo1g&t=2508s)
Volvo Cars' machine learning clusters were running on AWS, with Kubernetes clusters built using EKS, while the company also had Azure and GCP infrastructure. Leonard says staying cloud agnostic is a strategic choice because data and workloads are distributed, and the company wants to avoid vendor lock-in. A central DevOps team maintains the shared cloud-native stack, including Kubernetes, Istio, Argo CD, and related services. Leonard's team owns the machine learning layer, while teams developing individual systems should own their systems. The platform aims to provide common patterns without preventing specialized solutions for manufacturing or deployment into cars.

## Notable quotes
- Leonard Aukea: "We need to do the same for our machine learning systems." (27:29)
- Leonard Aukea: "You can't be an AI expert these days if you don't have some grounding in software engineering." (24:08)
- Leonard Aukea: "We want to treat machine learning like we treat our cars." (27:29)
- Leonard Aukea: "We don't want to force data scientists to write YAML and become DevOps engineers." (54:34)
- Leonard Aukea: "We're trying to build kind of an open source mindset, but within the company." (51:12)

## Tools & references mentioned
- Volvo Cars
- Kubernetes
- Kubeflow
- Tecton
- GitHub
- GitLab
- Harbor
- LakeFS
- DVC
- KServe
- Thanos
- Prometheus
- Grafana
- Arize
- Feast
- EKS
- AWS
- Azure
- GCP
- Istio
- Argo CD
- Martin Zinkevich
- Jeremy Jordan
- Amundsen
- Kafka

## Who should watch
- You are building an internal machine learning platform and need a practical pattern for connecting repositories, pipelines, images, access controls, and project templates.
- Your team is deciding how much platform infrastructure to centralize while leaving individual machine learning teams room to own their systems.
- You want to introduce software testing, model regression tests, and code review to data science teams without turning every data scientist into a DevOps engineer.

## Editor's note

Leonard Aukea says Kubeflow lacked the CI/CD layer Volvo Cars needed, so the team connected Git repositories to pipeline automation, testing, and deployment. ZenML tracks each pipeline run's steps, inputs, outputs, and code version, so teams can keep reproducible workflows without rebuilding that record across infrastructure. Its stack configuration also lets the same pipeline code run on different platforms.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)
- [MLOps Insights](https://mlopstalks.com/talks/mlops-insights) (David Aponte-Demetrios Brinkmann-Vishnu Rachakonda, 37:47)
- [MLOps in Practice: Common Challenges and Lessons Learned](https://mlopstalks.com/talks/mlops-in-practice-common-challenges-and-lessons-learned) (Marouen Hizaoui & Mo Basirati, Machine Learning Reply, 1:03:01)
- [Operationalize Machine Learning at Scale with MLOps](https://mlopstalks.com/talks/operationalize-machine-learning-at-scale-with-mlops) (Christopher Bergh, DataKitchen, 57:50)
- [What MLOps Has Taught Me](https://mlopstalks.com/talks/what-mlops-has-taught-me) (Ewan Nicolson, Forecast, 54:14)