Evaluation gives teams a way to measure an LLM application's performance before changing it and to check whether those changes helped or hurt.
2
Public LLM leaderboards can reward overfitting to open test sets and metrics that do not match human judgment, so teams need private tests based on their own data distribution.
3
Fine-tuning changes how a model follows instructions, while RAG supplies external information, so the two techniques address different problems and can be used together.
Summary
Shahul Es explains evaluation as measuring and quantifying a system before iterating on it. He is skeptical of public LLM leaderboards because models can be fine-tuned on open test sets, while many metrics correlate poorly with human judgment, especially for long-form answers. Benchmarks can provide a rough comparison, but an application needs tests drawn from its own data distribution and dimensions that matter to its users. Shahul describes a process of creating a representative test set, choosing aspects such as fluency and whether a summary captures the source, then finding ways to score them. LLM-based evaluation can reduce annotation work, although factuality remains difficult. He recommends isolating the source of failures before changing prompts, retrieval, the base model, or fine-tuning. The conversation also separates RAG from fine-tuning. RAG brings in private facts, while fine-tuning teaches a model to follow instructions or produce a particular form of output. Shahul connects these ideas to his work on Ragas and Open Assistant.
Evaluation starts with measurement before iteration
Shahul defines evaluation as measuring and quantifying how a system performs. This applies to LLMs and traditional machine-learning systems. He says improvement requires a baseline first, because teams need to see whether an iteration had a positive or negative effect. Evaluation therefore supports repeated changes to a system rather than a one-time model comparison. Demetrios summarizes the practical implication as, "what you can't measure you can't actually fix," and Shahul agrees with the need to measure before deciding what to change.
Public leaderboards can reward models that fit the leaderboard
Shahul says open LLM leaderboards can become less reliable because teams optimize for a particular competition instead of building a generally useful model. When datasets and tasks are public, developers can fine-tune on them, which breaks the assumption that a test set contains examples the model has not seen. He compares this with Kaggle competitions, where a public test set gives feedback while a private test set determines the final result. The private portion limits direct optimization against the visible evaluation.
Metrics often miss what people care about in long-form answers
Shahul separates short-form and long-form evaluation. Answers such as "New York City" and "NYC" can be compared with relatively simple methods. Summaries and stories require dimensions such as coherence, fluency, and how well the output captures the source. He argues that traditional metrics for these dimensions often have a weak relationship with human judgment. A benchmark can offer a ballpark view of model performance, but it cannot guarantee how a model will behave on a particular application's data.
An application's test set should match its real data distribution
For an LLM application, Shahul recommends starting with a reasonable test set that reflects the prompts users actually send. A retrieval-augmented system focused on document summaries should include prompts about summarizing documents, rather than relying on unrelated benchmark tasks. The next step is to identify the aspects users care about, such as fluency or how much relevant information a summary retains. Those aspects then need a scoring method, which could use string matching or another LLM. The evaluation should run whenever a pipeline component changes.
LLM judges can reduce annotation work, with limits around factuality
Shahul describes using an LLM to evaluate another LLM, including multimodal systems. He warns that the judge can prefer the style of another model instead of assessing the dimensions people actually need. In RAG systems, one useful dimension is groundedness: whether an answer stays within the supplied context. An LLM can help quantify this and detect hallucination without a fully annotated golden answer. Shahul is more cautious about factuality. Deciding whether a statement is factually correct remains difficult, and he compares it with the long-standing problem of fake-news detection.
Debugging should isolate the failing component before changing it
When an application performs poorly, Shahul recommends isolating the error first. The cause could be the base LLM, its training data, the prompts, retrieval, or another pipeline component. He gives Llama 2 and coding as an example. If the model saw too few code tokens during pre-training, fine-tuning may improve results but can also overfit. Understanding what the model was trained on helps determine whether a prompt change, a different model, or fine-tuning is a sensible next step.
Fine-tuning changes instruction-following rather than adding private facts
Shahul says fine-tuning is useful when a model needs to think or respond in a particular way. His example is function calling, where the model learns to map a request such as asking for the weather in Boston to a function argument containing Boston. Fine-tuning can improve how the model follows instructions and formats its output. It should not be treated as a way to add a company's factual knowledge. Fine-tuning a model on private enterprise information is prone to overfitting, so RAG remains the appropriate way to supply that information.
RAG and fine-tuning solve different problems and can be combined
Shahul rejects the idea that fine-tuning and retrieval-augmented generation are competing replacements. RAG retrieves information that the model did not learn during pre-training, while fine-tuning teaches the model how to use instructions or perform a particular task. A model can therefore be fine-tuned for a desired behavior and still use RAG for current or private facts. He also connects model quality to data quality through his Open Assistant work, saying that preparing and filtering high-quality fine-tuning data is harder than running a fine-tuning job on random examples.
"Fine tuning doesn't really make sense in that situation because by fine tuning you cannot really inject new factual information to the model."Shahul Es38:48
Who should watch
You are building an LLM or RAG application and need a practical starting point for creating tests from real user data.
Your team is comparing open-source models through public benchmarks and wants to understand why leaderboard results may not predict production behavior.
You are deciding whether a poor result calls for better retrieval, prompt changes, a different base model, or fine-tuning.
