# Navigating via Retrieval Evaluation to Demystify LLM Wonderland

Atita Arora, Qdrant | AI in Production 2024 | 12:53

Source: https://www.youtube.com/watch?v=XPfd3u3YfkA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/navigating-via-retrieval-evaluation-to-demystify-llm-wonderland
Published: 2024-04-04
Tags: evals, multimodal, rag, reranking

## TL;DR
- Retrieval quality directly affects the quality of responses produced by RAG systems.
- Teams can improve retrieval with better chunking, model fine-tuning, query rewriting, and changes to retrieval methods.
- Evaluation datasets can mislead when they contain bias, narrow queries, redundant documents, or too much personalization.

## Summary
Atita Arora argues that retrieval deserves close attention in LLM applications because RAG systems depend on the information they retrieve before a model generates an answer. She reviews common retrieval metrics, including precision, recall, F1, average precision, DCG, NDCG, MRR, similarity scores, and human evaluation. She then connects poor retrieval with poor responses and hallucinations. Arora discusses ways to improve retrieval, including chunking strategies, model fine-tuning, query rewriting, and changes to how systems interact with vector databases. She also covers retrieval for images, audio, and video. The final part focuses on evaluation problems. Biased test data, narrow query sets, redundant documents, and the lost-in-the-middle problem can make a system appear more effective than it is. Personalization can also push users into an echo chamber or filter bubble. Her main practical point is to evaluate the retrieval layer carefully before judging the generated answer.

## Key ideas
### Information retrieval shapes everyday AI experiences
[02:58](https://www.youtube.com/watch?v=XPfd3u3YfkA&t=178s)
Arora says information retrieval is the first step behind applications such as online shopping, report building, assistants, films, and music. The quality of retrieved information can affect what people buy, how they work, and the media they consume. She argues that relevance has become harder to measure because structured and unstructured data keeps growing, alongside images, video, and audio. The information shown to a user is also subjective, contextual, and temporal, so a retrieval system needs evaluation that accounts for more than a single universal notion of relevance.

### Retrieval quality determines much of a RAG system's output
[06:00](https://www.youtube.com/watch?v=XPfd3u3YfkA&t=360s)
Arora describes RAG as a process with retrieval, augmentation, and generation. The system retrieves information from a knowledge store, adds it to a prompt, and sends that prompt to a large language model. Her warning is direct: "if you retrieve garbage" the system can produce hallucinations. She says a RAG system is as good as its retrieval, and gives the contrasting case that bad retrieval leads to bad responses while good retrieval leads to good responses. This makes retrieval evaluation an essential part of assessing the complete application.

### Standard retrieval metrics answer different evaluation needs
[04:41](https://www.youtube.com/watch?v=XPfd3u3YfkA&t=281s)
Arora lists precision, recall, F1, and average precision as basic metrics for retrieval quality. When ranking matters, she recommends measures such as DCG, NDCG, and MRR. Similarity scores are also used in vector search, while human evaluation is expensive and tends to be left until the final stage. Her list shows that evaluation depends on what the application needs, especially whether it must place relevant results near the top or assess the quality of a broader ranked list.

### Several retrieval changes can improve results
[07:10](https://www.youtube.com/watch?v=XPfd3u3YfkA&t=430s)
Arora says chunking strategy is usually a good place to start when improving retrieval. Where resources allow, model fine-tuning can also help. She mentions query rewriting and adapted retrievers as other approaches. Teams are also changing how they retrieve from and interact with vector databases. Her point is that retrieval quality depends on choices across the pipeline, including how content is split, how queries are reformulated, which model is used, and how the database is searched.

### Multimedia retrieval needs content-specific methods
[08:21](https://www.youtube.com/watch?v=XPfd3u3YfkA&t=501s)
For images, Arora distinguishes metadata-based retrieval from content-based image retrieval. Content-based systems analyze visual features such as colour, texture, shape, and spatial arrangement, then index images for retrieval by visual content rather than text annotations. For audio, she says many customers use Whisper-generated transcripts and process them as text. Video is harder because it combines audio and images, so retrieval methods need to combine both approaches.

### Evaluation metrics also apply to multimedia relevance
[09:09](https://www.youtube.com/watch?v=XPfd3u3YfkA&t=549s)
Arora names mean average precision, similarity score, and AUC as metrics used in these settings. She explains that AUC measures classifier performance, with higher values indicating better performance. A value of one is perfect, while 0.5 corresponds to random decisions. The ROC curve plots sensitivity, or true positive rate, against the false positive rate. In her example, this helps identify whether an image or other visual content is relevant.

### Biased evaluation data can hide retrieval failures
[10:27](https://www.youtube.com/watch?v=XPfd3u3YfkA&t=627s)
Arora warns that training and testing data can bias both models and retrieval evaluation. If the test set lacks diversity, its metrics may reward a system for retrieving one narrow type of document. A collection dominated by similar or redundant documents can make retrieval look effective even when it performs poorly across a wider range of content. Queries that focus too narrowly on one domain create a similar problem because they do not test whether the system can retrieve diverse, relevant information in other contexts.

### Long context and personalization create additional risks
[10:49](https://www.youtube.com/watch?v=XPfd3u3YfkA&t=649s)
Arora connects large context windows with the lost-in-the-middle problem, where information in the middle of a long context may receive less attention. She also warns that personalization can become excessive. A system may place users in an echo chamber or filter bubble when it repeatedly returns similar information. Evaluation therefore needs to test how retrieval behaves across different users, queries, document types, and positions within the context.

## Notable quotes
- "The key piece here is retrieval, which is why it is very important to pay attention to the retrieval quality." (06:00)
- "If you retrieve garbage, I mean you're going to get hallucinations." (06:21)
- "Your RAG system is as good as your retrieval." (06:21)
- "I think that's where usually you should begin if you have resources available, you could also try model fine-tuning." (07:35)

## Tools & references mentioned
- Qdrant
- Demetrios Brinkmann
- Whisper
- RAG
- retrieval-augmented generation
- DCG
- NDCG
- MRR
- AUC
- ROC curve

## Who should watch
- You are building a RAG application and need to decide what to measure before tuning the language model.
- Your retrieval results look plausible, but you do not know whether chunking, ranking, or query rewriting is the main problem.
- You are creating an evaluation set and need to account for bias, narrow queries, redundant documents, multimedia content, or personalization.

## Related talks

- [Supercharging Your RAG System: Techniques and Challenges](https://mlopstalks.com/talks/supercharging-your-rag-system-techniques-and-challenges) (Tengyu Ma, Voyage AI and Stanford University, 40:20)
- [Embeddings and Retrieval for LLMs: Techniques and Challenges](https://mlopstalks.com/talks/embeddings-and-retrieval-for-llms-techniques-and-challenges) (Anton Troynikov, Chroma, 35:19)
- [RAG from Scratch with Best Practices](https://mlopstalks.com/talks/rag-from-scratch-with-best-practices) (Skylar Payne, 1:29:52)
- [Evaluating and Integrating ML Models](https://mlopstalks.com/talks/evaluating-and-integrating-ml-models) (Morgan McGuire & Anish Shah, Weights & Biases, 51:57)
- [The Real E2E RAG Stack](https://mlopstalks.com/talks/the-real-e2e-rag-stack) (Sam Bean, Rewind.ai, 1:10:07)