# MLflow 3.0: The Future of AI Agents

Eric Peter, Databricks | AI Agent Builders Summit | 12:51

Source: https://www.youtube.com/watch?v=ROlUmzOIQ6A
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mlflow-3-0-the-future-of-ai-agents
Published: 2025-07-24
Tags: agents, evals, governance, observability

## TL;DR
- Agent development adds software engineers, open-ended language evaluation, domain-expert review, tracing, and guardrails to the familiar machine learning lifecycle.
- An agent operations platform needs access to language models, end-to-end observability, framework integrations, open standards, and enterprise security and governance.
- MLflow 3.0 adds tracing, customizable GenAI evaluation, prompt and agent versioning, labeling workflows, and integrations across open-source MLflow and Databricks.

## Summary
Eric Peter compares the traditional MLOps lifecycle with the development lifecycle for GenAI applications and agents. Training is often replaced by prompting, tuning, tool use, and function calling. Evaluation becomes harder because users ask the same thing in different ways and models return free-form language. Teams need traces, human feedback, language-model judges, and a way to turn production traffic into test cases. Peter describes five requirements for agent operations: access to LLMs, shared observability across development and production, integrations with authoring frameworks, open standards, and enterprise security. MLflow 3.0 extends MLflow with tracing, prompt and agent versioning, customizable evaluation, review workflows, and an LLM gateway. The Databricks-hosted edition adds governance, data intelligence features, and connections to lakehouse workflows. The talk gives a product-focused view of how one platform can cover classic ML and GenAI applications together.

## Key ideas
### Agent development brings software engineers into ML application work
[02:24](https://www.youtube.com/watch?v=ROlUmzOIQ6A&t=144s)
Eric Peter says software engineers who have no machine learning background are now building GenAI applications. The workflow still includes familiar stages such as evaluation, deployment, and data preparation, but the development step has changed. Training is often replaced by prompting, tuning, tool use, and function calling. Teams also need to trace what happens inside the application so they can understand its behavior. This makes agent development closer to software development in some respects, while adding model-specific concerns that ordinary application teams may not have handled before.

### Open-ended language makes GenAI evaluation different from classic testing
[02:39](https://www.youtube.com/watch?v=ROlUmzOIQ6A&t=159s)
GenAI applications accept free-form inputs and produce free-form text. Peter gives the example that "What's a healthy snack?" and "food to keep me fit" can express the same intent. Two answers can also differ in wording while having the same meaning. Traditional unit and integration testing assumes that a program's output can be labeled deterministically as right or wrong. GenAI evaluation therefore needs semantic judgments, often from LLM judges or human reviewers, rather than only exact comparisons or fixed expected outputs.

### Useful evaluation starts with debugging, feedback, and real traffic
[04:35](https://www.youtube.com/watch?v=ROlUmzOIQ6A&t=275s)
Peter says teams need to inspect complex applications, collect feedback from experts, and measure open-ended language outputs. They also need to turn real user traffic into test cases. A small collection of invented prompts cannot show how people actually use an application. Evaluation must be customized for the business because no general-purpose evaluator will work well for every use case. MLflow's approach includes traces, review and labeling workflows, customizable Python metrics, and configurable LLM judges.

### Agent operations needs five platform capabilities
[05:36](https://www.youtube.com/watch?v=ROlUmzOIQ6A&t=336s)
Peter defines agent operations as a standardized process from development through production. He names five requirements: access to LLMs, end-to-end observability that works in development and production, integrations with the SDKs and authoring frameworks teams use, support for open standards, and enterprise security and governance. The last category includes encryption, audit logs, and network controls. Open standards matter because teams should be able to change vendors without being locked into one protocol.

### MLflow 3.0 connects agent workflows with existing MLflow primitives
[07:11](https://www.youtube.com/watch?v=ROlUmzOIQ6A&t=431s)
MLflow 3.0 extends capabilities added for GenAI applications and agents while keeping them on the same primitives as classic MLflow. Peter describes an open-source product built on OpenTelemetry and compatible with LLM and agent frameworks. It includes tracing, trace visualizations, offline and online evaluation, prompt and agent versioning, and a gateway that provides one interface across LLM providers such as OpenAI and Anthropic. This lets enterprises manage classic ML and GenAI through a shared lifecycle and governance model.

### Tracing and evaluation support repeated improvement
[09:51](https://www.youtube.com/watch?v=ROlUmzOIQ6A&t=591s)
Peter says MLflow tracing can provide visibility into an application with one line of code, including inputs, outputs, assessments, and feedback. Teams can search, filter, and query traces to find cases that need improvement, then analyze those records in a SQL environment. The GenAI evaluation command includes judges for common cases, while allowing teams to supply Python code as custom metrics and adapt LLM judges to their own requirements. Teams can compare data and prompt versions through the product interface.

### The Databricks edition adds governance and lakehouse workflows
[08:53](https://www.youtube.com/watch?v=ROlUmzOIQ6A&t=533s)
The Databricks-hosted version includes the open-source capabilities plus enterprise governance features such as access controls, permissions, and audit logs. Peter also describes Databricks Data Intelligence features built around LLM judging and evaluation. Since GenAI applications take data in and produce data out, the hosted product connects those records to lakehouse workflows, where teams can build reports, dashboards, and other downstream uses.

## Notable quotes
- "The step of training has largely been replaced with tuning or prompting or choose your favorite buzzword." (02:42)
- "It's not sufficient to just be able to make up a few queries. You actually have to see how users use your application in order to test and evaluate them." (05:13)
- "You don't want to be locked into one vendor's protocol." (06:33)
- "One line of code and you'll instantly get full visibility into what's happening inside your application." (09:51)

## Tools & references mentioned
- MLflow
- Databricks
- OpenTelemetry
- OpenAI
- Anthropic
- MLOps
- GenAI
- agent ops
- LLM judges
- SQL

## Who should watch
- You are building a GenAI application and need to evaluate free-form outputs, inspect traces, or collect review from domain experts.
- Your team wants one lifecycle and governance approach for traditional ML systems and agent applications.
- You are comparing open-source tooling with a Databricks-hosted platform and need to understand the differences in evaluation, security, and data workflows.

## Related talks

- [MLflow Leading Open Source](https://mlopstalks.com/talks/mlflow-leading-open-source) (Databricks' Corey Zumar, 58:24)
- [Big updates to MLflow 3.0](https://mlopstalks.com/talks/big-updates-to-mlflow-3-0) (Eric Peter & Corey Zumar, Databricks, 29:31)
- [GenAI in production with MLflow](https://mlopstalks.com/talks/genai-in-production-with-mlflow) (Ben Wilson, Databricks, 19:53)
- [Evaluation-Driven Development with MLflow 3.0](https://mlopstalks.com/talks/evaluation-driven-development-with-mlflow-3-0) (Yuki Watanabe, Databricks, 28:13)
- [Making MLflow](https://mlopstalks.com/talks/making-mlflow) (Corey Zumar, Databricks, 59:11)