# Evaluating LLM applications A pack of 8 sessions from the MLOps Community YouTube channel, in the order to watch them. 6h 11m of video. Page: https://mlopstalks.com/packs/evaluating-llm-applications A prompt change looks better on your favorite examples but annoys real users. Your judge awards high scores to answers reviewers reject. A model upgrade fixes one task and quietly damages another. Begin with the outcome the application should help someone achieve, then build examples that reflect its actual use. Separate that evaluation set from the benchmarks and examples you have been optimizing against. The next talks turn quality into checks, examine misleading judge scores and calibrate automated decisions against human judgment. Keep feedback tied to the version and run it describes. End with a staged release process that makes competing quality measures visible. The aim is a repeatable comparison you can act on, rather than a single reassuring number. ## This pack is for you if - You cannot tell whether a prompt or model change improved the application overall. - Your automated judge disagrees with the people who understand the task. - Your evaluation set contains familiar demos but misses the failures users report. ## The talks, in order ### 1. How Product Metrics Become LLM Evaluations Raza Habib, Humanloop | 53:07 | MLOps Community Video: https://www.youtube.com/watch?v=KWcE8ybs09A Summary: https://mlopstalks.com/talks/how-product-metrics-become-llm-evaluations.md Why first: Habib starts with what a user wanted to accomplish, including the cases where no single reference answer exists. That gives you a reason to measure a particular property. Before choosing a judge or collecting a benchmark, decide which outcome your development score is supposed to approximate. ### 2. All About Evaluating LLM Applications Shahul Es, Exploding Gradients | 50:40 | MLOps Podcast Video: https://www.youtube.com/watch?v=LOpv3vQeLxU Summary: https://mlopstalks.com/talks/all-about-evaluating-llm-applications.md Why second: Public leaderboard performance can reward familiarity with the test rather than usefulness on your task. Shahul makes the case for examples from the application's own distribution. With a target outcome established, this is the check that your eventual score will describe the people and requests you actually serve. ### 3. Evaluating LLM-based Applications Josh Tobin, Gantry | 49:50 | LLMs in Production 2023 Video: https://www.youtube.com/watch?v=r-HUnht-Gns Summary: https://mlopstalks.com/talks/evaluating-llm-based-applications.md Why here: Tobin shows how to grow a small evaluation set from difficult inputs and choose metrics according to the evidence available. His grammar-correction example catches an unwanted translation behavior, turning an observed mistake into a criterion for the next comparison. This makes the dataset an evolving record of requirements rather than a fixed collection of demos. ### 4. Systematically Test and Evaluate Your LLMs Apps Gideon Mendels, Comet | 1:01:43 | MLOps Podcast Video: https://www.youtube.com/watch?v=WhNozRPwgiY Summary: https://mlopstalks.com/talks/systematically-test-and-evaluate-your-llms-apps.md Why here: Exact assertions still matter, but equivalent answers should not fail because their wording differs. Mendels combines hard rules with semantic comparisons and model judges, then records the configuration behind each experiment. That makes the comparisons repeatable; the next two talks ask whether the judges themselves deserve your trust. ### 5. LLM Evaluation with Arize AI's Aparna Dhinakaran Arize AI's Aparna Dhinakaran | 55:48 | MLOps Podcast Video: https://www.youtube.com/watch?v=00AQjBDM46Q Summary: https://mlopstalks.com/talks/llm-evaluation-with-arize-ais-aparna-dhinakaran.md Why here: A number from one to ten can look precise without measuring what its label promises. Dhinakaran's examples motivate explicit classifications and checks tailored to the actual failure. Narrowing the question makes the automated verdict easier to interpret before you test its agreement with expert reviewers. ### 6. Holistic Evaluation of Generative AI Systems Jineet Doshi, Intuit | 56:04 | MLOps Podcast Video: https://www.youtube.com/watch?v=VJ0k0C1mGdg Summary: https://mlopstalks.com/talks/holistic-evaluation-of-generative-ai-systems.md Why here: Doshi treats the judge as something that also needs validation. Expert-created examples, simple scoring scales and detailed grading notes help expose disagreement instead of averaging it away. This adds a calibration step to the metric choices above, including checking that agreement holds over time. ### 7. Evaluation-Driven Development with MLflow 3.0 Yuki Watanabe, Databricks | 28:13 | MLOps Community Video: https://www.youtube.com/watch?v=7Q2Z9CYvdRc Summary: https://mlopstalks.com/talks/evaluation-driven-development-with-mlflow-3-0.md Why here: Watanabe connects assessments to the exact trace, reviewer and version they describe. His 2025 demonstration shows how production failures can become evaluation cases without losing that context. Once the judging method is credible, this record lets another person reproduce a comparison and understand why a result changed. ### 8. Evaluating Quality and Improving LLM Products at Scale Austin Bell, Slack | 15:11 | AI in Production 2024 Video: https://www.youtube.com/watch?v=F5t2XJaSpVY Summary: https://mlopstalks.com/talks/evaluating-quality-and-improving-llm-products-at-scale.md Why last: Bell gives the comparison a release decision. Small prototype sets lead to broader validation and then product experiments. His preprocessing example improves formatting while reducing another quality measure, showing why a team must weigh the whole result. Finish by making that tradeoff explicit instead of promoting whichever version wins one score.