Quality is the main barrier to putting agentic applications into production because agent behavior is unpredictable and quality has several dimensions.
2
Evaluation-Driven Development uses data collection, building, human feedback, automated evaluation, and production monitoring as a repeating development process.
3
MLflow 3.0 connects traces, feedback, evaluations, lineage, datasets, and monitoring so teams can track agent versions and investigate failures.
Summary
Yuki Watanabe presents Evaluation-Driven Development as a way to build agentic applications with more confidence. He compares it with test-driven development, with evaluations providing the anchor for an agent whose behavior can change across prompts, tools, models, and user inputs. The process starts with collecting data, building and debugging the agent, gathering human feedback, running automated evaluations, and monitoring production behavior. MLflow 3.0 supports this process through tracing, one-line integrations, trace annotations, sharable sessions for nontechnical reviewers, built-in and custom judges, experiment lineage, production monitoring, and dataset management. Watanabe also explains how traces connect evaluation results to inputs, outputs, model versions, parameters, and intermediate tool calls. He is clear that general-purpose judges have limits and that teams often need custom criteria. In the Q&A, he says MLflow tracks agent versions and behavior but does not provide the agent's memory or dictate its architecture.
Watanabe says a survey of ML practitioners, data scientists, and agent builders found that quality was the number one organizational blocker to launching agents in production. Agent behavior is difficult to assess because language models are nondeterministic, applications accept free-form input, and quality includes several dimensions beyond accuracy. A single agent can also contain an LLM, search, and tools, so changing one part can break another. These properties make it difficult to rely on the fixed checks used in many traditional machine learning systems.
Evaluation-Driven Development makes evaluation the anchor
Watanabe compares Evaluation-Driven Development with test-driven development. The process begins with collecting useful data, then building and iterating on the agent. Human evaluation remains necessary for checking whether the core behavior is correct, even though it takes time. Automated evaluation helps scale checks across many agent versions. After deployment, monitoring continues because the project can experience regressions, changing traffic, or concept drift. The process is cyclical rather than something that ends at the first production launch.
Traces expose the intermediate steps of an agent run
MLflow traces capture the execution of an agent, including intermediate LLM calls, tool calls, inputs, outputs, metadata, and available tools. Watanabe shows a trace for a LangGraph agent, with the execution displayed as a tree and the selected step's details shown alongside it. This lets developers inspect what happened inside a framework call, find exceptions, and understand the data passed between steps. The same trace data can later support evaluation and monitoring. MLflow's integrations produce OpenTelemetry-compatible traces, so they can also be used with other observability platforms.
Human feedback needs to be stored with its context
Manual evaluation often creates spreadsheets containing feedback from different people, criteria, and agent versions. MLflow lets reviewers annotate feedback directly on traces. An assessment can record whether an answer is correct, whether it matches the expected context, or whether the agent selected the right tools. The feedback stores information about who added it, the rationale, and changes made later. Watanabe also describes annotation sessions, which let domain experts review selected traces through a shared UI without learning MLflow or receiving access to other models, runs, or metrics.
LLM-as-a-judge can automate feedback, but Watanabe says the judge itself is another agent that needs development. Instructions must produce consistent and accurate feedback. MLflow's evaluation APIs include built-in judge metrics and allow custom judges written in natural language. Results are logged with lineage connecting prompts, parameters, models, and runs. Teams can also define a scorer function with access to inputs, outputs, traces, and ground-truth labels. That supports project-specific checks, such as extracting retrieved documents from a trace and passing them to a precision or recall calculation.
Production monitoring applies the same scorers to real traffic
Static evaluation datasets cannot represent all of the dynamic inputs an agent receives in production. Watanabe says MLflow 3.0 with the Databricks platform can monitor production traces after tracing and storage are configured. Teams can compute operational metrics such as latency, errors, and token counts, as well as custom quality metrics. Results appear in dashboards, and users can open the traces behind a problematic chart segment to reproduce and investigate the issue. The same scorers can be saved and loaded so offline evaluation and production monitoring use a shared approach.
Production traces can become a growing evaluation dataset
Watanabe describes dataset management as the final part of the process. Teams can start with a small manually created or publicly available dataset, then add cases when users report incorrect answers. MLflow can create datasets from written inputs and outputs or by selecting actual traces. Production traces with user feedback can therefore become evaluation examples. Connecting these datasets to an automated evaluation step lets a team check that a model update fixes a reported issue without creating the same mistake again.
MLflow tracks agent development without defining agent architecture
In the Q&A, Watanabe says MLflow is not intended to provide agent memory or prescribe a particular agent framework. Its role is to track versions, runs, parameters, traces, and evaluation results, including agents that use their own memory mechanisms. He describes agent development as exploratory, with paths that are less fixed than traditional training and validation workflows. Evaluation gives that exploration an anchor. He also says fine-tuning and prompt optimization are areas where MLflow connects traditional machine learning workflows with agent development.
