# How Product Metrics Become LLM Evaluations

Raza Habib, Humanloop | MLOps Community | 53:07
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=KWcE8ybs09A
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-product-metrics-become-llm-evaluations
Published: 2025-06-03
Tags: evals, human-in-the-loop, observability, testing

## TL;DR
- The best evaluations approximate the outcome a user wants, so product metrics and evaluations should be closely related.
- Teams improve LLM systems by tracing production behavior, running test cases in development, and iterating on prompts, tools, models, and retrieval settings.
- Domain experts need to help define quality because engineers alone may not know what a good legal, language-learning, payroll, or support response looks like.

## Summary
Raza Habib argues that product metrics and LLM evaluations should measure nearly the same thing: whether a user achieved the outcome they wanted. Since generative systems often have no single correct answer, teams need proxies such as user feedback, corrections, next actions, custom evaluators, and human judgment. Humanloop developed from production feedback and prompt comparisons into a platform with tracing, observability, development datasets, evaluators, and versioned prompt and tool configuration. Raza describes a feedback loop in which teams inspect failures, form a hypothesis, change a prompt or system component, rerun evaluations, and deploy when the metrics meet their thresholds. He also explains why product managers, data scientists, and domain experts should work with engineers. The talk covers customer-facing use cases in legal technology, coding, education, and new agent applications. Raza ends with Humanloop's plans for agent evaluation and automated system optimization driven by evaluations and human feedback.

## Key ideas
### Evaluations should approximate the user outcome
[00:00](https://www.youtube.com/watch?v=KWcE8ybs09A&t=0s)
Raza says the ideal evaluation would measure whether the user achieved the goal they wanted. That might mean using a generated text, taking the next action, receiving a helpful support answer, or having an agent complete a task in the expected way. Development evaluations are proxies because the real user outcome is not available before production. For subjective tasks such as meeting summaries, there is no single correct answer. A sales call summary may need to preserve different information from another type of conversation, and different users may care about different details.

### Production feedback comes from several kinds of user behavior
[02:28](https://www.youtube.com/watch?v=KWcE8ybs09A&t=148s)
Raza describes three common baseline signals. Thumbs-up and thumbs-down feedback is useful, although only a small and extreme subset of users tends to provide it. Corrections show both how much a user changed an output and what the acceptable answer looked like in that case. The natural next action also provides a signal, such as whether someone sends a message or copies generated text. Teams can add use-case-specific evaluators for tone, brand fit, or other subjective properties, then run them in development, production, or both.

### LLM applications need tests because their behavior varies
[21:00](https://www.youtube.com/watch?v=KWcE8ybs09A&t=1260s)
Raza rejects the idea that evaluation is only a workaround for weak models. Even an extremely capable model still needs the right context, must meet the application's definition of success, and can change when a provider releases a new version. Different models also have different styles and capabilities. He compares this with software testing and machine learning test sets. The tests are different because the outputs are often subjective and stochastic, but teams still need to measure whether changes improve the system.

### Tracing connects production failures to concrete system changes
[16:48](https://www.youtube.com/watch?v=KWcE8ybs09A&t=1008s)
Humanloop's production tracing records the path of an agent, including the user's request, conversation turns, and actions taken by tools. Evaluations and feedback can be attached to those traces. During development, teams run the application over a dataset, score the outputs, inspect failures, and form a hypothesis about the cause. They might change a prompt, improve retrieval, modify a tool definition, or fine-tune a model. They then rerun the evaluation to see whether the change improved the measured results.

### Prompt configuration belongs outside application code
[25:03](https://www.youtube.com/watch?v=KWcE8ybs09A&t=1503s)
Raza says prompts and tool definitions should be separated from the code so they can be changed by the people who understand the domain. Humanloop tracks versions, serialized files, audit logs, and who changed each item. Teams can also synchronize those files with Git. A production application can request the currently deployed prompt or tool configuration, while a nontechnical user can edit the configuration through the platform. This gives prompt changes a review and history process similar to code changes.

### Quality needs engineers, product managers, data scientists, and domain experts
[20:23](https://www.youtube.com/watch?v=KWcE8ybs09A&t=1223s)
Engineers build the calls, pipelines, and application logic, but they may not know what good looks like in a legal or customer-support workflow. Product managers can often translate a product specification into evaluation criteria. Lawyers, language-learning specialists, real-estate experts, and support staff can judge whether outputs are acceptable. Data scientists add statistical discipline, such as separating held-out data and running stochastic systems multiple times. Human pass-fail judgments and critiques can later be distilled into an LM-based evaluator if it correlates well with human decisions.

### Reliable customer-facing systems create direct business value
[27:47](https://www.youtube.com/watch?v=KWcE8ybs09A&t=1667s)
Raza sees savings in customer-support automation, where better agents can increase deflection and reduce the time handled by people. He also describes revenue-generating applications in legal technology, coding tools, and education. Filevine has released AI products for contract and legal workflows, while companies such as Duolingo and McMillan are building learning products. Customer-facing applications need more reliability and confidence than many internal tools, which makes tracing and evaluation more important.

### Agent evaluations require coverage of branching behavior
[45:31](https://www.youtube.com/watch?v=KWcE8ybs09A&t=2731s)
Raza says agents are harder to evaluate because they can take many different paths and interact with users or external systems. A useful evaluation may need to mock a user who answers questions or a third-party API called by a tool. Humanloop is working on an agent builder and improved evaluation support for these systems. He also describes an auto-optimization prototype that proposes changes, reruns evaluations, examines failures, and tries further changes. Human feedback remains necessary because the system cannot infer every user preference without input.

## Notable quotes
- Raza Habib: "I think that there's no real difference between product metrics and evals or at least like the best evals are the same as product metrics." (00:00)
- Raza Habib: "What eval do is they allow you to get quantitative feedback on every change that you make." (09:44)
- Raza Habib: "If you don't have good evals, I don't think you're going to be able to get very far on the other pieces." (10:20)
- Raza Habib: "Writing a spec is not that different to writing eval." (20:38)
- Raza Habib: "We're not magically able to build a better system than you like without your input." (51:33)

## Tools & references mentioned
- Humanloop
- Filevine
- Duolingo
- Gusto
- Fundrise
- Windmill
- McMillan
- Harvey
- Entra
- Ironclad
- Cursor
- Windsurf
- Claude Code
- OpenAI
- Anthropic
- GitHub Copilot
- Dario Amodei
- DSPy
- OpenTelemetry
- Jira
- Linear
- Netflix
- TikTok
- ChatGPT
- Claude

## Who should watch
- You are building an LLM product and need a practical way to connect user feedback, production traces, and development tests.
- Your prompts or tool definitions are spread through application code, and product or domain specialists cannot change them without engineering help.
- You are designing an agent and need to account for branching behavior, external tools, mocked users, and human judgments.

## Related talks

- [Evaluating LLM-based Applications](https://mlopstalks.com/talks/evaluating-llm-based-applications) (Josh Tobin, Gantry, 49:50)
- [Evaluation](https://mlopstalks.com/talks/evaluation-panel-1-large-language-models-in-production-conference-part-2) (Abi Aryan, Independent Consultant & Amrutha Gujjar, Structured & Josh Tobin, Gantry & Sohini Roy, NVIDIA, 38:19)
- [From Building Self-driving Cars to Building LLM Applications](https://mlopstalks.com/talks/from-building-self-driving-cars-to-building-llm-applications) (Effy Zhang, Baserun, 10:45)
- [Evaluating Quality and Improving LLM Products at Scale](https://mlopstalks.com/talks/evaluating-quality-and-improving-llm-products-at-scale) (Austin Bell, Slack, 15:11)
- [Systematically Test and Evaluate Your LLMs Apps](https://mlopstalks.com/talks/systematically-test-and-evaluate-your-llms-apps) (Gideon Mendels, Comet, 1:01:43)