# Why is MLOps Hard in an Enterprise?

Maria Vechtomova & Basak Eskili, Ahold Delhaize | MLOps Podcast | Episode 159 | 55:06
Hosted by Abi Aryan

Source: https://www.youtube.com/watch?v=RKbMww5kxHE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/why-is-mlops-hard-in-an-enterprise
Published: 2023-05-30
Tags: cost, model-serving, monitoring, platform-teams

## TL;DR
- Enterprise MLOps works better when a central team provides a simple golden path built from tools the organization already uses.
- Ahold Delhaize shares model code across brands while keeping data, training, and brand-specific configuration separate.
- MLOps adds data access, data quality, model monitoring, and cost management problems to the software delivery problems found in DevOps.

## Summary
Maria Vechtomova and Basak Eskili describe how Ahold Delhaize standardizes machine learning across 19 brands. The brands have different levels of data science capability, yet many need similar products such as cross-sell recommendations, demand forecasts, and personalized offers. The team started with a maturity assessment, interviews, and existing tools. Databricks, GitHub, GitHub Actions, Kubernetes, and Azure became parts of a simple golden path rather than a large collection of new tools. The brands share Python packages, deployment workflows, and model logic, while each brand keeps its own data and trains its own model. The engineers also moved deployment ownership closer to the data science teams, which removed the slow handoff between data scientists and DevOps engineers. Maria and Basak are candid about remaining gaps, especially monitoring and cloud cost visibility. They describe a 90 percent cost reduction after replacing oversized, unmanaged Kubernetes deployments with more consistent workflows and better cluster choices.

## Key ideas
### Production experience pushed both engineers toward MLOps
[05:20](https://www.youtube.com/watch?v=RKbMww5kxHE&t=320s)
Maria Vechtomova started as a data analyst almost nine and a half years earlier, writing R scripts and scheduling jobs on a server at KPN. She moved into data science, built APIs, and began deploying models herself because nobody else was available to do it. She later built a Model Factory several times with different tools. Basak Eskili came from data science and saw notebooks being treated as products while projects remained at the MVP stage because nobody could operationalize them. She moved into machine learning engineering to close the gap between data scientists and development engineers.

### A golden path hides enterprise infrastructure from data scientists
[09:09](https://www.youtube.com/watch?v=RKbMww5kxHE&t=549s)
Maria describes the central framework as a golden path for bringing models to production. Data scientists repeat much of the same operational work, such as putting scripts somewhere, connecting version control, and setting up deployment. The central team provides reusable workflows so a data scientist can get a project running in about five minutes without handling service users, service principals, or other infrastructure details. Basak adds that the standard solution should stay simple. The team chooses the needed components instead of combining many tools with overlapping functions.

### The deployment pattern changes with the model product
[11:52](https://www.youtube.com/watch?v=RKbMww5kxHE&t=712s)
Ahold Delhaize uses two broad pipeline patterns. Batch models run preprocessing, train on a schedule, and create predictions that are delivered to another system. Demand forecasts are one example. Other models run behind APIs, such as the cross-sell service that suggests products while a customer is shopping online. Kubernetes provides a standardized API deployment process across brands, while Databricks serverless API deployment is being used to simplify it further. The model still needs tuning for each brand because the data differs, but deploying the same model structure to another brand takes only a few minutes.

### Sharing code lets brands reuse models without sharing data
[20:20](https://www.youtube.com/watch?v=RKbMww5kxHE&t=1220s)
Ahold Delhaize has 19 brands, and some have data science teams while others have none. The brands often need similar products, so the team designs models to be configurable and parameterized. For the cross-sell model, a central repository contains the model logic as a Python package. Each brand has its own repository, data location, configuration, hyperparameters, and separately trained model. The brands share code rather than data or model artifacts. This structure lets one model design be reused while preserving each brand's data boundaries.

### The maturity assessment started with product-level questions
[15:44](https://www.youtube.com/watch?v=RKbMww5kxHE&t=944s)
Before choosing the stack, the team assessed MLOps maturity at the product level. The questionnaire asked whether a model used version control and whether a deployment could be traced to its code, model artifact, and data. Maria says the assessment had 70 questions. The team interviewed people in scheduled calls, and some filled in an Excel sheet shared through SharePoint. The first results were uncomfortable because the highest score was around 60, but the exercise gave teams a concrete view of their practices. Maria also mentions a standards document and a 20-page data science methodology document.

### Deployment ownership reduced the wall between teams
[30:40](https://www.youtube.com/watch?v=RKbMww5kxHE&t=1840s)
Before the new process, external companies sometimes sent zipped code or wheel files by email to a DevOps team. The two teams often used different environments, so errors were sent back and forth and could take weeks to resolve. Maria says getting the permissions needed for direct deployment took a year. Basak describes the main change as breaking the wall between data scientists and DevOps engineers. The machine learning team can create and deploy models, so ownership stays closer to the people who understand the model and its code.

### MLOps has data and model problems that ordinary DevOps does not
[33:47](https://www.youtube.com/watch?v=RKbMww5kxHE&t=2027s)
Maria says the data part creates a major difference between MLOps and DevOps. Code can stay unchanged while data changes and causes errors, so access to production data in development environments becomes an important issue. Deployed models also need quality checks because a model can deteriorate while software health checks continue to pass. The team monitors business behavior as well as technical operation. Maria gives an example of a cross-sell model that repeatedly suggested trash bags without anyone noticing for a long time. The result can damage customer trust even when the financial cost is limited.

### Reusable templates help data scientists write production code
[40:31](https://www.youtube.com/watch?v=RKbMww5kxHE&t=2431s)
The team lets data scientists begin with notebooks, especially in Databricks where they can access data directly. It encourages them to move functions and classes outside the notebook, create Python packages early, and write modular, production-ready code. Databricks Repos, GitHub synchronization, and the Databricks VS Code extension support that transition. A cookiecutter template creates a repository with permissions, a simple main file, deployment code, and a continuous integration pipeline that checks code quality. The workflow also includes unit testing and documentation requirements, and deployment can be blocked when the code does not meet the checks.

### Cost management exposed waste in the old deployment process
[46:08](https://www.youtube.com/watch?v=RKbMww5kxHE&t=2768s)
Cloud cost became one of the team's remaining work areas. The team tags Databricks jobs so Azure cost management can associate resources with projects and brands, although it still wants a dashboard combining run metadata with cost. Maria describes an earlier process that created an oversized Kubernetes cluster for a job and then deleted it. Failed jobs could leave clusters running because there was no monitoring, creating unnecessary expense. The standardized MLOps framework reduced those costs by 90 percent. The team is also moving from interactive clusters to job clusters and reviewing Azure virtual machine sizes.

## Notable quotes
- Maria Vechtomova: "It's not about the tools, but it's also how you use the tools." (27:48)
- Maria Vechtomova: "We don't share data, we don't share models. The only thing we share is the code." (20:20)
- Basak Eskili: "The biggest change we made with Maria is to actually break the wall between data scientists and DevOps engineers and get the responsibility of deploying models ourselves." (31:15)
- Maria Vechtomova: "The data part makes all the difference here." (33:55)
- Maria Vechtomova: "By doing the MLOps in this standardized way, using the framework that we built, we were able to reduce the cost by 90 percent." (50:13)

## Tools & references mentioned
- Ahold Delhaize
- KPN
- Model Factory
- Databricks
- GitHub
- GitHub Actions
- Jenkins
- Kubernetes
- AWS
- SageMaker
- Azure
- Spark
- Argo CD
- Databricks Repos
- VS Code
- Grafana
- Prometheus
- Azure Data Explorer
- Elastic
- ChatGPT
- GitHub Copilot
- IKEA

## Who should watch
- You work in a large company where brands or business units already have different ML stacks and need a shared delivery process.
- Your data scientists can build models but hand deployment to another team, creating long feedback loops and unclear ownership.
- You need to assess MLOps maturity, reuse model code across teams, or understand where cloud spending is coming from.

## Editor's note

Maria Vechtomova says Ahold Delhaize gives data scientists a golden path that gets a project running in about five minutes without infrastructure details. ZenML lets teams write pipelines as Python steps and run the same code on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud provider's services by changing the configured stack. Each run records its steps, inputs, outputs, and code version.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Operationalize Machine Learning at Scale with MLOps](https://mlopstalks.com/talks/operationalize-machine-learning-at-scale-with-mlops) (Christopher Bergh, DataKitchen, 57:50)
- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)
- [The Motivation for MLOps](https://mlopstalks.com/talks/the-motivation-for-mlops) (Steven Fines, CoreLogic, 56:42)
- [MLOps: Isn't That Just DevOps?](https://mlopstalks.com/talks/mlops-isnt-that-just-devops) (Ryan Dawson, Seldon, 1:06:32)
- [MLOps at Volvo Cars](https://mlopstalks.com/talks/mlops-at-volvo-cars) (Leonard Aukea, Volvo Cars, 57:31)