# Making MLflow

Corey Zumar, Databricks | MLOps Coffee Sessions | Episode 103 | 59:11
Hosted by Mihail Eric

Source: https://www.youtube.com/watch?v=odEWCeYPZkU
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/making-mlflow
Published: 2022-06-17
Tags: developer-experience, experiment-tracking, open-source

## TL;DR
- MLflow began by addressing the gap between narrowly focused machine learning tools through open APIs and an extensible core.
- MLflow's tracking API became its most widely used foundation because it records parameters, metrics, artifacts, source code, environments, and model metadata.
- The project is moving toward more complete machine learning workflows while trying to keep its APIs open, composable, and compatible with third-party tools.

## Summary
Corey Zumar explains how MLflow started at Databricks as an open-source response to a fragmented machine learning tooling ecosystem. Existing platforms often focused on one framework, such as TensorFlow or PyTorch, or handled only tracking or deployment. MLflow instead began with small APIs for tracking, models, and projects, then expanded through user feedback and adoption. Corey describes how Databricks customer usage helped the maintainers understand real workflows, while open-source users contributed issues, pull requests, integrations, and ideas. He discusses the tension between keeping components independent and building a more complete end-to-end platform. The conversation also covers authentication, multi-tenancy, feature stores, project-level abstractions, time-series workflows, and MLflow Pipelines. Corey is candid that MLflow's flexibility can leave teams unsure how to structure complex workflows. The maintainers are considering more opinionated paths while continuing to support external tools and community-led development.

## Key ideas
### MLflow started because machine learning tools were fragmented
[00:00](https://www.youtube.com/watch?v=odEWCeYPZkU&t=0s)
Corey says MLflow began after the team identified a gap in the MLOps space. Some platforms supported only TensorFlow models, others focused on PyTorch, metadata tracking, or deployment. There was little glue between these tools, and no broadly usable platform that covered the machine learning lifecycle across varied frameworks and software. A small team of nine Databricks developers built MLflow as an open-source platform around open APIs. Corey joined when the project was still an alpha release and watched its adoption grow far beyond what he expected.

### A small extensible API made the first MLflow release useful
[02:20](https://www.youtube.com/watch?v=odEWCeYPZkU&t=140s)
Corey draws a lesson from Spark's open-source development. Machine learning and ETL have many tools and user requirements, so building a separate solution for every niche would be difficult to maintain. MLflow instead started with thin interfaces. The tracking API could log a parameter, metric, or tag, then create a run or experiment. Early versions did not even include search. MLflow Models captured the model version and software environment, while Projects followed a similar minimal design. This let users try the system and give feedback without waiting for a complete platform.

### Tracking became MLflow's foundation because model work needs history
[05:34](https://www.youtube.com/watch?v=odEWCeYPZkU&t=334s)
Corey says the extensible tracking component has produced the most success. Teams need to know whether a model trained today is better than one trained last week, and that requires more than a single score. Tracking can include parameters, complex performance data, artifacts, source files, model objects, software environments, the person who built a model, and the system that produced it. Corey says the model registry, MLflow Models, and Projects are also used, but many integrations connect back to tracking. The API started small and expanded as users exposed new needs.

### Databricks usage and open-source feedback created the product feedback loop
[07:22](https://www.youtube.com/watch?v=odEWCeYPZkU&t=442s)
The maintainers used surveys soon after the alpha release to learn what users liked, found confusing, or wanted improved. MLflow's connection to Databricks added another source of information because customers used largely comparable functionality. Corey says observing hundreds or thousands of customers helped the team understand adoption and real workflows. He also treats issues and pull requests as useful signals. An issue means someone spent more than a few minutes with the software, while a pull request usually shows deeper investment. Organizational adoption and continued use matter more to the team than download counts or GitHub stars.

### MLflow separates open-source functionality from Databricks-specific features
[10:15](https://www.youtube.com/watch?v=odEWCeYPZkU&t=615s)
Corey says the relationship between MLflow and Databricks has changed over time. During the first year, open-source MLflow received features before Databricks. The team now asks whether a capability is useful to any data scientist or machine learning engineer, or whether it exists because of a Databricks-specific workflow. The first type is a candidate for open-source work and community RFCs. The second may belong in the Databricks product. Corey says the team still tries to open-source as much broadly useful functionality as possible while addressing customer needs quickly.

### Ease of adoption comes from file storage and independent components
[19:31](https://www.youtube.com/watch?v=odEWCeYPZkU&t=1171s)
Corey says MLflow avoids forcing users to install and connect a database, tracking server, and other services before they can begin. The core tracking system can write to a file system, so a user can start logging without setup. MLflow Models also stays close to the native framework. Once someone has installed TensorFlow or PyTorch, they can save an MLflow Model without adding many extra dependencies. Tracking, models, and the registry should each work independently. That lets a team adopt one component first and consider the rest later, while also making isolated testing easier for maintainers.

### MLflow wants broader coverage, but its open APIs must remain inclusive
[22:39](https://www.youtube.com/watch?v=odEWCeYPZkU&t=1359s)
Corey says the team has debated whether MLflow should become a complete end-to-end MLOps platform. He believes it needs to cover more of the lifecycle, including data ingestion, data management, feature work, monitoring, CI/CD, and web hooks. Customers often reach these problems after succeeding with tracking and deployment. The difficulty is that each area contains many opinionated tools and users often request different workflows. Corey says MLflow must create APIs that capture common patterns without excluding existing tools. The team has moved slowly in some areas because the requirements are still changing.

### MLflow Pipelines could make flexible workflows easier to structure
[45:13](https://www.youtube.com/watch?v=odEWCeYPZkU&t=2713s)
Corey says companies often struggle with how to organize MLflow rather than with infrastructure. A time-series forecasting team might train hundreds or thousands of models, retrain them hourly, and log each one separately. Later, it becomes difficult to know what was trained when or retrieve the best model for each category. MLflow's open and unopinionated design gives teams freedom, but it also leaves them to invent workflow conventions. The MLflow Pipelines RFC aims to provide more opinionated guidance about what to log and how to access it, creating a clearer path toward production.

## Notable quotes
- Corey Zumar: "There didn't really exist a platform that anybody could just pick up and use and would be compatible with the variety of tools and software that they needed." (00:55)
- Corey Zumar: "Instead of trying to purpose build solutions for every single niche tool and use case, it's often easier to create a really extensible and coherent core API and then let developers bring their own workflows to that API." (03:16)
- Corey Zumar: "One really nice thing about the core portion of MLflow is it's designed to be compatible with any host that has a file system." (20:25)
- Corey Zumar: "If they're filing pull requests that usually indicates an even more substantial level of investment." (18:57)
- Corey Zumar: "Push as much as you can into SQL, because that's just such a heavily optimized ecosystem with 50 plus years of investment." (54:25)

## Tools & references mentioned
- MLflow
- Databricks
- Spark
- TensorFlow
- PyTorch
- MLflow Plugins
- MLflow Models
- MLflow Pipelines
- Kubeflow
- Airflow
- K-Serve
- DataRobot
- H2O
- SageMaker
- Hermione
- Philosophy of Software Design
- ZenML

## Who should watch
- You are choosing between a flexible MLOps platform and a more opinionated end-to-end product, and you want to hear how an MLflow maintainer thinks about that trade-off.
- Your team uses MLflow tracking at scale and needs to understand why experiment organization, model retrieval, and workflow conventions become difficult.
- You maintain or build an open-source machine learning platform and want practical examples of API design, community feedback, compatibility, and commercial sponsorship.

## Editor's note

Corey Zumar says companies often struggle with how to organize MLflow, especially when a forecasting team trains hundreds or thousands of models and later needs to find what ran when. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace a model or artifact back to its producing data and code.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLflow Leading Open Source](https://mlopstalks.com/talks/mlflow-leading-open-source) (Databricks' Corey Zumar, 58:24)
- [MLOps Engineering Labs Recap, Part 1](https://mlopstalks.com/talks/mlops-engineering-labs-recap-part-1) (John Savage, Overstock & Alexey Naiden & Varuna Jayasiri & Michel Vasconcelos, Bank of Nordeste, 59:26)
- [MLflow Pipelines: Opinionated ML Pipelines in MLflow](https://mlopstalks.com/talks/mlflow-pipelines-opinionated-ml-pipelines-in-mlflow) (Xiangrui Meng, Databricks, 49:15)
- [MLOps Engineering Labs Recap, Part 2](https://mlopstalks.com/talks/mlops-engineering-labs-recap-part-2) (Laszlo Sranger & Artem Yushkovsky, Neuro & Paulo Maia, Nilgai, 1:04:16)
- [MLflow vs Kubeflow 2022](https://mlopstalks.com/talks/mlflow-vs-kubeflow-2022) (Byron Allen, Contino, 1:05:40)