# MLOps with Databricks

Maria Vechtomova, Ahold Delhaize | Marvelous MLOps | MLOps Podcast | Episode 314 | 52:44
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=Oa6qZPlOv3c
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mlops-with-databricks
Published: 2025-05-13
Tags: build-vs-buy, data-quality, deployment, model-serving

## TL;DR
- Databricks can replace a collection of separately managed MLOps tools when a company already uses the platform for data engineering and has accepted it as a vendor.
- Maria recommends packaging ML code and using Databricks Asset Bundles instead of treating notebooks as production code or relying on difficult local approximations of Databricks runtimes.
- Databricks is a poor fit for some serving workloads, especially low-latency Kubernetes applications or systems that exceed its workspace request capacity.

## Summary
Maria Vechtomova explains why Databricks is often a practical MLOps choice for large companies. A platform reduces the work of combining tools, and vendor approval can matter as much as technical capability. She prefers Databricks to the Azure components available in her organization because it is easier to use for ML. Her criticism is direct. Databricks development still revolves around notebooks, while production ML code should be packaged. She recommends Asset Bundles for deployment and development. Maria also describes problems with the Databricks feature store, including awkward feature functions, runtime-dependent behavior, limited data types, and lineage that does not carry through pandas workflows. Her own team trains models and runs much of its batch processing on Databricks, while using Azure Functions, databases, or Kubernetes where serving needs demand it. The conversation also covers MLflow, data quality ownership, failed data-contract efforts, her course, and her forthcoming book.

## Key ideas
### A platform can reduce the cost of assembling an MLOps stack
[03:42](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=222s)
Maria compares platforms with the earlier practice of assembling model registries, experiment tracking, artifact storage, serving, monitoring, and data versioning from separate tools. Individual tools may fit each component well, but connecting and operating them requires substantial investment and a large team. Cloud adoption adds vendor sourcing, security, compliance, and approval work. Maria says platforms have become good enough for most MLOps needs, especially when a company already uses one for data engineering. In her organization, Databricks was already available and was easier for ML than the Azure components they could have used.

### Databricks adoption follows existing data and vendor decisions
[08:33](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=513s)
Maria says Databricks became a common choice because data engineering already happens there and data is stored in Unity Catalog. Adding ML becomes a logical extension of that setup. She also points to the difficulty of bringing new vendors into a large organization. Demetrios describes choosing an inferior tool because it had already been accepted, avoiding a long onboarding process. Maria says Databricks is unusually open to criticism. She believes its developers often know about missing or awkward features, even when those fixes are not their first priority.

### Notebook-centered development makes production ML harder
[06:18](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=378s)
Maria's strongest criticism is the path from a notebook to production. She began as a data scientist and used notebooks because that was how Python and data science were commonly taught. When she first built a production endpoint, she saw how difficult it was to turn notebook work into deployable code. She argues that ML code should be packaged. Databricks runtimes make local reproduction difficult because a local environment is only an approximation of the managed runtime. Her advice is to teach data scientists how to write production-ready code or translate notebook work into that form.

### Asset Bundles provide a better development and deployment path
[15:02](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=902s)
Maria recommends Databricks Asset Bundles as an underused way to work with the platform. A bundle can define a workflow in JSON or Databricks YAML and deploy the workflow together with its packages, Python files, and other required assets. Before Asset Bundles, her team built internal deployment logic to upload and coordinate those pieces. Maria now sees the feature as useful for development as well as deployment. She says many people use bundles for deployment but miss their value during development.

### The Databricks feature store has awkward behavior and limits
[17:11](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=1031s)
Maria describes feature lookup and feature functions as the main ways to interact with the Databricks feature store. A lookup returns values for a key, but it has no fallback when the key is missing. A feature function can provide fallback logic, yet it requires defining a Python function in SQL. Maria objects that the function has no directly associated code version and can behave differently across Python versions and runtimes. Feature specs also execute their components without conditional statements, and feature functions can fail with complex output types. Feature engineering code is tied to the Databricks environment, while lineage works with Spark but stops working after conversion to pandas.

### Serving should follow latency, scale, and existing infrastructure
[26:20](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=1580s)
Maria would not recommend Databricks model serving for a website running on Kubernetes when low latency requires the model to run in the same Kubernetes environment. She says Databricks would be slower in that setup. She also gives a workspace-wide serving limit of 20,000 requests per second under stated assumptions, with a lower practical capacity possible. For recommendation systems, she describes precomputed results stored in a database and retrieved through a FastAPI application, Azure Functions, or another service. MLflow serving can also run on Kubernetes. Databricks is more attractive when its inference tables simplify endpoint monitoring.

### Maria's real stack mixes Databricks with other services
[39:50](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=2390s)
Maria's organization uses Databricks for model training, Spark-based data processing, Databricks Workflows, and data in Unity Catalog. Another team owns raw data ingestion through Azure Data Factory, while Maria's team adds domain-specific transformations for personalization. Results are either written to a database for lookup through Azure Functions or deployed as a model through Databricks model serving. The team had wanted to use Databricks feature serving and online tables, but limitations prevented that. This left them with several pipelines instead of one combined Databricks deployment flow.

### Data quality depends on the consumer's requirements
[46:09](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=2769s)
Maria's team consumes data produced by another engineering team. The producer checks schemas, acceptable value ranges, and normal row counts. Maria's team checks statistical properties that matter to its models. Because other consumers do not need those checks, the producing team may miss problems that break personalization work. They tried to establish data contracts, but Maria says the effort did not succeed because the producer team's priorities and reporting structure were different. A consumer may need support from management to change another team's process.

### MLflow has expanded beyond its earlier, less intuitive experience
[34:52](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=2092s)
Maria says MLflow's documentation and usability have improved significantly. She mentions MLflow Traces, the prompt registry, and the MLflow Gateway as recent capabilities, especially for LLM work. She describes MLflow serving as an open-source option that can run on Kubernetes, with Databricks using the same underlying model-serving format while hiding deployment details. When Databricks serving fails, users can run the MLflow model locally to debug it. Maria compares the packaging idea to BentoML, where a model is placed in a format that can become a Docker image or be deployed through a platform.

## Notable quotes
- Maria Vechtomova: "I feel like I've seen it all, to a certain extent, like how do you connect different pieces of your ML setup to make sure it's robust and you can roll back things whenever needed." (04:05)
- Maria Vechtomova: "In my opinion notebooks harm the MLOps life cycle much more than anything else out there." (07:25)
- Maria Vechtomova: "If you want to have a professionally written ML code it must be packaged." (13:57)
- Maria Vechtomova: "I think everyone should use whatever makes sense for the situation you are in." (26:24)
- Maria Vechtomova: "If Kubernetes is a big thing of what you do, I would totally go for it and then deploy just FastAPI app and then look up in some Cosmos DB, DynamoDB, or whatever database you have." (41:18)

## Tools & references mentioned
- Marvelous MLOps
- Databricks
- Azure
- Azure Data Factory
- Unity Catalog
- MLflow
- MLflow Traces
- MLflow prompt registry
- MLflow Gateway
- BentoML
- Kubernetes
- FastAPI
- Spark
- PySpark
- pandas
- Polars
- Feast
- Hopsworks
- Tecton
- Azure Functions
- Cosmos DB
- DynamoDB
- MongoDB
- App Insights
- Prometheus
- Maven
- Riley

## Who should watch
- You are deciding whether a managed platform is worth the trade-offs of assembling open-source MLOps tools yourself.
- Your organization already uses Databricks for data engineering and you need practical guidance on packaging, deployment, serving, and monitoring ML workloads.
- You are considering the Databricks feature store or model serving and need to understand where Maria's team found limitations.

## Editor's note

Maria Vechtomova argues that ML code should be packaged because Databricks runtimes make it hard to reproduce and deploy notebook work locally. ZenML lets teams write workflows as Python steps and run the same pipeline code on a laptop or Kubernetes through a configured stack. Each run records the steps, inputs, outputs, and code version.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Databricks Model Serving V2](https://mlopstalks.com/talks/databricks-model-serving-v2) (Rafael Pierre, Databricks, 43:17)
- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)
- [Packaging MLOps Tech Neatly for Engineers and Non-engineers](https://mlopstalks.com/talks/packaging-mlops-tech-neatly-for-engineers-and-non-engineers) (Jukka Remes, Haaga-Helia University of Applied Sciences, 8wave AI, 55:31)
- [MLOps: Isn't That Just DevOps?](https://mlopstalks.com/talks/mlops-isnt-that-just-devops) (Ryan Dawson, Seldon, 1:06:32)
- [MLOps - The Blind Men and the Elephant](https://mlopstalks.com/talks/mlops-the-blind-men-and-the-elephant) (Saurav Chakravorty, Brillo, 55:02)