A prompt change looks better on your favorite examples but annoys real users. Your judge awards high scores to answers reviewers reject. A model upgrade fixes one task and quietly damages another. Begin with the outcome the application should help someone achieve, then build examples that reflect its actual use. Separate that evaluation set from the benchmarks and examples you have been optimizing against. The next talks turn quality into checks, examine misleading judge scores and calibrate automated decisions against human judgment. Keep feedback tied to the version and run it describes. End with a staged release process that makes competing quality measures visible. The aim is a repeatable comparison you can act on, rather than a single reassuring number.
Why first: Habib starts with what a user wanted to accomplish, including the cases where no single reference answer exists. That gives you a reason to measure a particular property. Before choosing a judge or collecting a benchmark, decide which outcome your development score is supposed to approximate.
Why second: Public leaderboard performance can reward familiarity with the test rather than usefulness on your task. Shahul makes the case for examples from the application's own distribution. With a target outcome established, this is the check that your eventual score will describe the people and requests you actually serve.
Why here: Tobin shows how to grow a small evaluation set from difficult inputs and choose metrics according to the evidence available. His grammar-correction example catches an unwanted translation behavior, turning an observed mistake into a criterion for the next comparison. This makes the dataset an evolving record of requirements rather than a fixed collection of demos.
Why here: Exact assertions still matter, but equivalent answers should not fail because their wording differs. Mendels combines hard rules with semantic comparisons and model judges, then records the configuration behind each experiment. That makes the comparisons repeatable; the next two talks ask whether the judges themselves deserve your trust.
Why here: A number from one to ten can look precise without measuring what its label promises. Dhinakaran's examples motivate explicit classifications and checks tailored to the actual failure. Narrowing the question makes the automated verdict easier to interpret before you test its agreement with expert reviewers.
Why here: Doshi treats the judge as something that also needs validation. Expert-created examples, simple scoring scales and detailed grading notes help expose disagreement instead of averaging it away. This adds a calibration step to the metric choices above, including checking that agreement holds over time.
Why here: Watanabe connects assessments to the exact trace, reviewer and version they describe. His 2025 demonstration shows how production failures can become evaluation cases without losing that context. Once the judging method is credible, this record lets another person reproduce a comparison and understand why a result changed.
Why last: Bell gives the comparison a release decision. Small prototype sets lead to broader validation and then product experiments. His preprocessing example improves formatting while reducing another quality measure, showing why a team must weigh the whole result. Finish by making that tradeoff explicit instead of promoting whichever version wins one score.