# Big updates to MLflow 3.0

Eric Peter & Corey Zumar, Databricks | MLOps Community | 29:31

Source: https://www.youtube.com/watch?v=2ABNr-IJNsM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/big-updates-to-mlflow-3-0
Published: 2025-11-07
Tags: debugging, evals, monitoring, observability

## TL;DR
- MLflow brings software-style testing, evaluation, and monitoring to generative AI applications, where inputs and outputs are free-form and often need human judgment.
- MLflow tracing, evaluators, prompt comparison, and production monitoring help teams find hallucinations, define quality criteria, and improve application behavior over time.
- Agentic Insights uses agents and MLflow traces to find root causes across many production requests, rank problems, and recommend which issues to fix first.

## Summary
Eric Peter and Corey Zumar explain how MLflow applies an engineering lifecycle to generative AI applications. Eric starts with tracing, which captures the steps inside an application, user feedback, retrieved data, and model calls. He then demonstrates plain-language evaluation criteria, evaluation datasets, prompt comparison, and judges that run across sampled production traffic. The examples involve generating sales emails and checking their accuracy, groundedness, sentiment, and relevance. Corey then presents Agentic Insights for debugging multi-agent systems. A simple LLM judge usually sees only selected fields such as a request and response. An agentic judge can inspect the whole trace, follow tool calls, and identify failures such as malformed database queries. A coding assistant connected to MLflow through MCP can analyze many traces, test hypotheses, rank recurring problems, and write a report with supporting evidence. The proposed workflow reduces the manual work involved in finding and prioritizing quality issues.

## Key ideas
### MLflow treats generative AI quality as a lifecycle problem
[01:48](https://www.youtube.com/watch?v=2ABNr-IJNsM&t=108s)
Eric Peter compares generative AI development with the software development lifecycle. Software teams commonly test code locally, write unit and integration tests, run QA, and observe production behavior before shipping. Generative AI needs a similar process, but its inputs and outputs are free-form. Users can change what they ask without warning, and the application returns text that may need human judgment. MLflow adds offline evaluations, CI/CD-based evaluations, and online monitoring so teams can check quality before release and continue checking it in production.

### Tracing shows what happened inside an AI application
[05:57](https://www.youtube.com/watch?v=2ABNr-IJNsM&t=357s)
Eric demonstrates an application that generates sales emails from CRM information. MLflow tracing can instrument the application with one line of code and expose the retrieved company data, data processing steps, and model calls. User feedback is attached directly to the trace, so a developer can see that a user marked an email as bad without manually matching feedback to logs. The trace gives the team a concrete place to investigate when an answer is wrong.

### Plain-language guidelines turn business requirements into evaluations
[07:43](https://www.youtube.com/watch?v=2ABNr-IJNsM&t=463s)
The demo uses MLflow SDKs to create an LLM judge. Eric enters business criteria as plain-language guidelines, such as requiring meeting discussions to be summarized with the same sentiment. MLflow evaluates traces against those guidelines and shows why a result failed. In the example, the model hallucinates product features after seeing information in retrieved data. The team can save failing traces in an evaluation dataset and reuse them while changing the application.

### Prompt versions can be compared against the same failures
[09:29](https://www.youtube.com/watch?v=2ABNr-IJNsM&t=569s)
Eric reruns evaluation on two prompt versions using a dataset of companies with previous problems. The revised prompt adds rules intended to prevent hallucinations. MLflow compares the runs side by side, allowing the developer to see changes in accuracy and groundedness. Eric says the demo skips the many intermediate prompt versions that a real iteration would usually involve, but the same comparison workflow supports repeated testing.

### Production judges can annotate sampled traffic automatically
[10:48](https://www.youtube.com/watch?v=2ABNr-IJNsM&t=648s)
After a judge is working, Eric says it can be registered and started to evaluate production traffic at a chosen sampling rate. This extends the same guideline-based checks from offline experiments into online monitoring. MLflow can work with LLM SDKs, authoring frameworks, plain Python, and TypeScript. The Databricks version also includes governance, production-scale operation, and access to traces in the data warehouse.

### Agentic judges inspect complete traces instead of selected fields
[15:56](https://www.youtube.com/watch?v=2ABNr-IJNsM&t=956s)
Corey Zumar explains that a conventional LLM judge usually receives a few extracted fields, such as an agent's query and response, and scores one narrowly defined metric. It cannot inspect the full sequence of tool calls and intermediate outputs. MLflow's proposed agent-as-a-judge can navigate the complete trace, inspect operations dynamically, and answer broader questions such as whether tools were called correctly or unnecessarily. In the telco example, it finds malformed SQL queries and failed tool calls.

### Coding assistants can rank recurring failures across traces
[21:01](https://www.youtube.com/watch?v=2ABNr-IJNsM&t=1261s)
Agentic Insights connects a coding assistant to MLflow traces through MCP. The assistant first infers what the agent is built to do from its traces. It then forms hypotheses about problems, searches other traces to confirm or reject them, and produces a ranked report with evidence. In the demo, tool call failures account for 40% of the issues, while latency varies from 7 seconds to 3.5 minutes. The report recommends fixing database query failures and handling missing information gracefully.

### Quality criteria depend on business input and governance
[27:23](https://www.youtube.com/watch?v=2ABNr-IJNsM&t=1643s)
In the questions, Eric says developers need agreement with business stakeholders about what quality means. MLflow's human labeling capabilities are intended to let business users inspect and interact with traces during that process. For governance, he points to MLflow on Databricks and its integration with Unity Catalog. The accuracy criteria in the email example were entered by the team, although MLflow also provides some criteria out of the box.

## Notable quotes
- Eric Peter: "With MLflow what we wanted to do was help bring this life cycle to GenAI." (03:49)
- Eric Peter: "MLflow tracing gives you the ability to essentially put one line of code to instrument your application and be able to see what's happening under the hood." (05:57)
- Corey Zumar: "Our solution is agent as a judge." (17:15)
- Corey Zumar: "You've just automated the entire root cause and prioritization analysis process." (25:25)

## Tools & references mentioned
- Databricks
- MLflow
- MLflow 3.4
- MCP
- Unity Catalog
- GPT-4o

## Who should watch
- You are building a generative AI application and need to connect business definitions of quality to repeatable evaluations.
- Your application produces traces, but developers still read them manually to find hallucinations, failed tool calls, or slow requests.
- You want to use an agent or coding assistant to investigate production behavior and rank which fixes deserve attention first.

## Related talks

- [MLflow 3.0: The Future of AI Agents](https://mlopstalks.com/talks/mlflow-3-0-the-future-of-ai-agents) (Eric Peter, Databricks, 12:51)
- [Evaluation-Driven Development with MLflow 3.0](https://mlopstalks.com/talks/evaluation-driven-development-with-mlflow-3-0) (Yuki Watanabe, Databricks, 28:13)
- [GenAI in production with MLflow](https://mlopstalks.com/talks/genai-in-production-with-mlflow) (Ben Wilson, Databricks, 19:53)
- [MLflow Leading Open Source](https://mlopstalks.com/talks/mlflow-leading-open-source) (Databricks' Corey Zumar, 58:24)
- [Making MLflow](https://mlopstalks.com/talks/making-mlflow) (Corey Zumar, Databricks, 59:11)