LLM applications need task-specific evaluations because general model benchmarks do not show whether an application works for its actual use case.
2
RAG systems need separate checks for retrieval relevance, grounding of the response in retrieved text, and correctness of the final answer.
3
Teams should deploy early, measure real outcomes, and try prompt and retrieval changes before treating fine-tuning as the next step.
Summary
Aparna Dhinakaran explains why evaluating an LLM application is different from evaluating a base model. Teams care about whether a particular prompt, retrieval setup, and application flow produce a useful answer for a specific task. LLM-as-a-judge can score hallucination, toxicity, correctness, and whether an answer is supported by retrieved information, although using the judge inline can add latency. In a RAG application, retrieval needs its own evaluation because bad context leads to bad answers. Aparna describes Phoenix, Arize AI's open-source product, as a way to trace the calls inside an LLM application and evaluate each stage. She also discusses research showing that numeric score evaluations can be misleading, so classification labels often work better. Her advice is to get an application into production, learn where it fails, and try simpler prompt and retrieval changes before fine-tuning or choosing a model mainly for future customization.
LLM applications need task evaluations rather than only model benchmarks
Aparna says teams deploying LLM applications face a large gap between a Twitter demo and a production system. Traditional ML has familiar classification, regression, and ranking metrics, while an LLM application may need to answer one specific kind of question correctly. General model evaluations, such as the leaderboards on Hugging Face, measure broad capability across tasks. Application teams usually care about how well a chosen model, prompt template, retrieval setup, and application structure perform on their own task. That makes task evaluations more useful than broad model scores for day-to-day development.
LLM-as-a-judge can evaluate answers without waiting for ground truth
Aparna describes LLM-as-a-judge as asking one LLM to evaluate the output of another. Teams can provide the user question, the relevant information retrieved by the system, and the final answer, then ask whether the answer responds to the question and is factually based on the supplied context. Common checks include hallucination, toxicity, correctness, and whether a chatbot says something inappropriate about its own product. This approach can help assess outputs when a ground-truth answer is not available. For higher-risk applications, the owner may block a response with a poor evaluation, while lower-risk systems may show it and use the result to improve the application.
RAG systems need separate evaluations for retrieval and generation
A RAG application can fail before the answer is generated. Aparna says teams should first ask whether the retrieved document or chunk is relevant to the user's question. They should then check whether the generated answer is based on that retrieved text and whether the answer itself is correct. Retrieval connects an LLM to private company data that was not part of the model's training data, so the quality of the retrieved context matters directly. Arize tested providers with a needle-in-a-haystack style setup, placing a value at different positions and context lengths before asking the model to find it. Aparna says GPT-4 performed best in that testing, while Mistral also impressed her among smaller model providers.
Prompt changes can materially alter retrieval results
The retrieval tests showed that some models changed their results substantially when the prompt changed by a sentence or two. Aparna says this makes prompt iteration an important part of evaluating retrieval. A model may appear capable in one prompt configuration and perform differently in another, even when the underlying task remains the same. Teams building private-data applications therefore need to test the prompts they actually use rather than rely on a provider's general reputation or a single benchmark result. The retrieval check should measure whether the system can find the relevant information inside the context window, including when the information appears early, late, or at different context lengths.
Phoenix traces the internal steps of an LLM application
Aparna describes Phoenix as Arize AI's open-source product for LLM observability and evaluation. A seemingly simple chatbot interaction can contain several calls: the user's question, embedding generation, retrieval, context synthesis, and response generation. Phoenix exposes the traces and spans for those calls, showing which steps took longer and which used more tokens. Teams can evaluate the final response and then inspect earlier spans when the answer is wrong. For a hallucinated response, they can evaluate the retrieval step and inspect whether each retrieved document was relevant. Phoenix also includes an evaluations library that can run in a notebook or pipeline and supports offline and online use.
Classification labels are often more meaningful than numeric LLM scores
Aparna discusses research comparing score evaluations with classification evaluations. A score evaluation asks an LLM to return a number, such as a frustration level from 1 to 10 or the percentage of words with spelling errors. Her tests found that the numeric output did not reliably correspond to the property being measured. A response could receive a high spelling-error score even when only a small share of the words had errors. She recommends explicit labels such as frustrated or not frustrated, with multiclass labels when needed, instead of assuming an LLM understands a smooth scale. Downstream systems that rely on an unreliable score can make poor decisions.
Custom evaluations must match the application's own failure modes
Aparna agrees that teams usually need to customize evaluations after starting with general templates. The relevant checks depend on what the application is meant to do and what kinds of failure matter to its users. Phoenix provides templates for areas such as hallucination, toxicity, and correctness, but teams can add their own evaluations. She expects this customization to matter more in LLM applications than in some traditional ML settings because the quality of an answer depends on the specific task, prompt, retrieved context, and output requirements.
Fine-tuning should come after simpler application changes
Aparna sees teams discussing fine-tuning before their applications are even deployed. She connects that instinct to traditional ML, where engineers train a model, inspect failures, and improve it with more data. In an LLM application, however, a prompt change or a retrieval change can sometimes produce very different results with less effort. Her advice is to start with the changes that can improve the application most easily before fine-tuning or training a model. She gives the same practical advice about open-source and private models: choose the model that produces the outcome the application needs, rather than choosing open source mainly because future fine-tuning sounds attractive.
Early production use exposes problems that offline work misses
Aparna agrees with the idea that teams should get an application into production quickly, then evaluate its outcomes. Demetrios Brinkmann adds that production reveals bottlenecks and failures that are hard to see beforehand. Aparna's closing advice is to evaluate outcomes, customize the evaluations, and understand the trade-offs between different evaluation types. Her point is practical: production use shows whether the prompts, retrieval, model, and application flow work for real users, while the evaluation results give the team evidence for what to change.
"First is, is the retrievable that was retrieved even relevant to the question asked, then was the output based on the retrieved text, and then, was the output itself answering correctly based on the information that was retrieved."Aparna Dhinakaran28:05
Who should watch
You are building a chatbot or RAG application and need to separate retrieval failures from answer-generation failures.
Your team is using LLM judges and wants to know when classification labels are safer than numeric scores.
You are considering fine-tuning or an open-source model before measuring what a simpler prompt or retrieval change would achieve.
