Evaluation helps teams decide whether an LLM application is ready for users and where its performance needs work.
2
LLM evaluations need data that resembles production use and metrics that predict the outcomes the product cares about.
3
The most practical workflow combines fast automated evaluation with carefully designed human checks.
Summary
Josh Tobin explains why evaluating LLM applications differs from evaluating traditional machine learning systems. With an off-the-shelf language model, the original training distribution is usually unavailable and production data already differs from it. Generative outputs also make simple metrics such as accuracy difficult to apply, while a single overall score can hide major differences across tasks. Tobin recommends building an evaluation set from real use cases, starting with a few interesting examples and adding difficult or unexpected production inputs over time. Metrics should depend on the task: exact correctness when there is a right answer, reference comparisons when there is a useful reference, and model-based or human judgments when outputs are open-ended. LLMs can evaluate other LLMs quickly, but they have biases. Human evaluation is slower and inconsistent, so Tobin argues for automated evaluations verified with targeted human checks. He demonstrates this process with a grammar-correction application in Gantry.
Evaluation gives teams a basis for release decisions and improvements
Tobin gives evaluation three practical jobs. It validates how the model performs, helps a team decide whether to release it to end users, and points to places where the system can improve. A useful evaluation should correlate with business outcomes. A single score can help machine learning systems optimize, but teams also need ways to slice performance across different kinds of inputs. Evaluation should run quickly and automatically enough that developers can use it after each change, while leaving room for human judgment where automation is not sufficient.
Traditional training, validation, test, and drift concepts fit LLMs poorly
In traditional machine learning, teams sample training and evaluation data from a training distribution, use a test set from production, and compare results to understand overfitting, domain shift, and drift. Tobin says this model breaks down for applications built with pretrained LLMs. Developers usually did not create the training data and cannot practically access the full distribution. Production tasks are also almost always different from the internet-scale data used to train the model, so distribution shift is already built into the problem.
Generative outputs make correctness and aggregate scores harder to define
For a cat-versus-dog classifier, the prediction can be compared with a known label and scored with accuracy. A generative application may produce several sentences that are all valid descriptions of the same input, which makes direct comparison less clear. A model can also perform well on some subjects and badly on others. Tobin uses the example of a system that is strong on startup questions, weaker on dogs and food, and very poor on physics. Whether that system is good depends on the product's users and tasks.
A useful evaluation needs representative data and a predictive metric
Tobin describes an evaluation as a dataset, one or more models, metrics applied to the outputs, and a report of the results. Evaluation data is useful when it resembles the inputs the application will see in production. A metric is useful when it predicts the outcome the product is trying to achieve, such as how people would rate an answer. Both parts matter. Even a metric that is well designed will provide little guidance if it runs on data unrelated to the application's real task.
Public benchmarks are informative only when their tasks match the application
Tobin ranks public benchmarks by what they measure. Functional correctness benchmarks are especially useful for code generation because generated code can be run against a task. Chatbot Arena gathers human preferences between model responses. Model-based evaluation asks one model to judge another. HELM and BIG-bench provide broad task comparisons, while older automated metrics that compare generated text with a gold answer often correlate poorly with human judgments. These benchmarks can provide context, but they usually do not measure a developer's specific use case or current prompting and fine-tuning methods.
Evaluation datasets should grow from small examples and production failures
Tobin recommends starting with a small, manually collected set rather than waiting for a perfect dataset. Developers can try a few inputs, keep examples that are difficult or reveal a possible user behavior, and run every prompt change against the growing set. An LLM can generate additional test inputs, although generated examples may lack the diversity a person would think of. Once users interact with the system, teams should add disliked outputs, outliers, underrepresented topics, and other difficult production examples. The dataset is an evolving record of use cases and failure modes.
Metric choice follows whether the task has an answer or a useful comparison
If a task has a correct answer, standard machine learning metrics may work. If there is no single correct answer but a reference exists, the output can be compared with that reference using semantic similarity or an LLM judgment about factual consistency. Developers can also compare two model outputs, check whether a new answer incorporates feedback on an old answer, verify output structure, or ask a model to assign a score. The choice depends on what evidence is available and what the application needs.
Model-based evaluators speed experiments but need bias checks
Using an LLM to judge another LLM allows more experiments to run without waiting for people. Tobin describes biases that include preference for particular score values, preference for the evaluator's own output, sensitivity to answer order, and preference for longer text. Human review is still needed before deployment and should be used to verify that the automated evaluator measures the intended property. Tobin's preferred arrangement is for developers to use automated evaluation during iteration, then spend human evaluation effort where it can increase confidence.
A production feedback loop makes evaluation part of development
Tobin compares the process with test-driven development. A team puts a basic model in production, gathers feedback, adds relevant examples to the evaluation set, and changes the prompt or model. It then runs automated or human evaluation before deploying the next version. In the grammar-correction demonstration, production inputs reveal that French sentences are being translated into English. The team adds those examples, changes the prompt to preserve the input language, adds that requirement as an evaluation criterion, and compares the new and old prompts across the full dataset.
"The first and probably most important quality of a good evaluation is it should be a number or a set of numbers that is highly correlated with the outcomes that you actually care about as a business."00:40
Who should watch
You are building an LLM application and need an evaluation set that reflects the inputs users will actually send.
Your team is relying on public benchmarks or informal prompt testing to decide whether a model is ready to ship.
You want to combine model-based grading with human review without making every experiment slow and expensive.
