Generative AI evaluation depends on the use case because open-ended outputs and broad model capabilities make universal metrics difficult.
2
A reliable evaluation program combines traditional NLP metrics, human review, red teaming, and model-based judges, with each method's limits understood.
3
RAG and agent systems need tests for individual components and the complete workflow, both before launch and continuously in production.
Summary
Jineet Doshi explains why generative AI evaluation remains difficult. Traditional machine learning models had narrow tasks, bounded outputs, and established metrics. LLMs can write, summarize, answer questions, and perform many other tasks, often with multiple valid outputs. He describes three evaluation families: traditional NLP methods and benchmarks, human evaluation, and LLMs acting as judges. Each has tradeoffs involving coverage, cost, scalability, bias, and domain knowledge. Doshi recommends calibrating judges against a trusted, expert-created dataset, using simple scoring scales, adding examples and grading notes, and monitoring judge drift. For RAG, teams should evaluate retrieval, chunking, embeddings, answer generation, and the end-to-end result. Agent evaluation adds tests for every step, tool choice, and tool parameters. Production evaluation uses explicit customer feedback, implicit behavior signals, and recurring checks. The discussion also stresses that system complexity raises the full cost of producing a quality answer.
LLMs make evaluation difficult because their outputs are open-ended
Jineet Doshi contrasts LLMs with classical machine learning models trained for narrow tasks. A classifier might output a probability, while a regression model produces a number, so metrics such as accuracy, precision, recall, and AUC fit the problem. An LLM can write a poem about PyTorch, summarize text, or answer questions, with many possible answers that could be acceptable. Doshi connects this to the broader difficulty of measuring intelligence. He says evaluation still depends on the use case, and there is no single right or wrong method for generative AI systems.
Evaluation has to cover the entire machine learning lifecycle
Doshi says evaluation should begin when a team chooses data and continue through model building, training, or fine-tuning. It remains necessary after deployment, when teams need to monitor quality and detect drift. He applies this to both classical machine learning and generative AI. At Intuit, he describes a large production environment that grew from single-digit models to thousands of models producing 58 billion predictions per day. That scale makes ongoing checks part of operating the systems, rather than a one-time activity before launch.
Traditional NLP methods work best when the task can be made more specific
One approach is to turn open-ended tasks into multiple-choice questions so that established metrics can be used. Doshi says this works for knowledge-based and reasoning-based tasks, but it does not cover many long-form or creative outputs. Another method compares a generated answer with a reference in embedding space. BLEU, BERTScore, cosine similarity, and edit distance can provide inexpensive comparisons, and semantic embeddings may support cross-language evaluation when they capture the meaning well. The limits are substantial. Domain-specific legal or medical language may require a suitable embedding model, and some metrics do not align reliably with human judgment.
Benchmarks need to match the product use case and avoid leakage
Doshi discusses benchmarks for knowledge, reasoning, physics, hallucination, and toxicity. Teams can combine several benchmarks to inspect different dimensions, but many rely on multiple-choice questions and miss open-ended behavior. Off-the-shelf benchmarks may be enough for some applications. A specialized product may need a custom benchmark. Doshi also warns that public benchmark data can enter model training data, producing inflated scores. He cites GLUE and MMLU as examples of benchmarks that had to be updated because of this type of problem.
Human evaluation gives broad coverage, although it is expensive and subjective
Manual labeling remains a strong option because people can judge quality, toxicity, bias, and other dimensions together. Doshi says evaluators need precise criteria, since one person may call an answer a three out of five while another calls it a five out of five. Pairwise preferences can be easier than absolute scores, with one answer judged better than another. Chatbot Arena uses this type of comparison to aggregate preferences into a leaderboard. Red teams extend human evaluation by training specialists to find ways to break models before customers encounter those failures. The approach becomes harder when legal, medical, or financial expertise is required.
LLM judges scale evaluation, but their judgments need calibration
Using a more capable LLM as a judge can provide some of the coverage of human review at greater scale. Doshi says results depend on the judge model and on prompt formatting. Judges can prefer verbose answers even when shorter answers are equally good, and they can show positional bias toward the first answer. A jury of several models or a second model that judges the first judge may reduce some bias, though these approaches add cost and create questions about how to weight the judgments. Doshi recommends calibrating a judge against a trusted dataset created by domain experts and manual reviewers, then checking its correlation over time.
Simple judge outputs and detailed grading notes improve model-based evaluation
Doshi recommends pairwise comparison for subjective tasks. When absolute scoring is needed, he prefers a low-precision scale such as zero or one over a range from zero to 100, because fine distinctions are difficult for judges to make consistently. Few-shot examples can give the judge a reference point. Asking for reasons makes failures easier to debug. He also describes grading notes, where each evaluation example includes detailed instructions about what to inspect. For a code execution task, notes might tell the judge to check columns, input formatting, null errors, and similar exceptions. These instructions are especially useful in domain-specific evaluations.
RAG and agent systems require component tests alongside end-to-end tests
For RAG, Doshi says teams should evaluate both individual parts and the complete system. Retrieval can use established search metrics such as reciprocal rank, nDCG, recall at K, and precision at K. Teams also need to test chunking strategies and embedding models, then assess answer generation and the final response from the user's query through the whole pipeline. Agents add more steps and tools. Each step output needs evaluation, as does the end-to-end result. For every tool call, teams must check whether the agent selected the right tool and passed the right parameters. More tools increase the evaluation burden.
"It's really important to have and like a good definition of all these different criteria to ensure consistency across human labelers."Jineet Doshi19:39
Who should watch
You are building an LLM application and need to choose between benchmarks, human review, red teaming, and LLM judges.
Your RAG or agent workflow has several components and you need a testing plan that covers retrieval, generation, tools, and the full path.
You operate generative AI in production and need to account for quality drift, customer feedback, judge calibration, and the full cost of producing an answer.
