# Is More Really Better: Delve Into Document Strategy | MLOps Community | 16:00 Source: https://www.youtube.com/watch?v=eEvramjyy-w Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/is-more-really-better-delve-into-document-strategy Published: 2024-12-17 Tags: evals, long-context, rag, reranking ## TL;DR - The best number of retrieved documents and in-context examples depends on the task, the document set, and the available context budget. - The proposed performance model uses task-specific informativeness, logarithmic gains, and an inverse sigmoid to predict how retrieval and demonstration settings affect results. - Adding documents improves recall, but relevance and ranking quality can lag behind and extra context can introduce noise. ## Summary This session examines how the number of retrieved documents, in-context examples, retrieval iterations, and context length affect RAG performance. The speaker explains a model that tries to predict the best configuration for a given budget. Its parameters include the number of documents, the number of demonstrations, and the number of retrieval-generation iterations. The model also estimates how informative documents and examples are for each task, since a useful document can produce a larger gain than a weak one. Validation suggests that the model can fit some settings, although its ability to generalize to unseen tasks remains unclear. Extrapolation works better between some context lengths than others, with transitions such as 32k to 128k proving difficult. The practical lesson is that retrieving more documents raises recall, but can add irrelevant material and noise. Better retrieval, ranking, and up-to-date knowledge bases remain necessary. ## Key ideas ### The best document and shot counts depend on the available budget [00:00](https://www.youtube.com/watch?v=eEvramjyy-w&t=0s) Holding the number of shots constant while increasing the number of documents produces different performance changes for different mechanisms. The same is true when the document count stays fixed and the number of shots changes. The practical question is how many shots and documents to show for a particular budget. The speaker describes the paper as looking for scaling laws that predict the best parameter set rather than assuming that more context always produces the best result. ### The model treats retrieved RAG examples as demonstrations [02:31](https://www.youtube.com/watch?v=eEvramjyy-w&t=151s) The discussed approach can place several complete RAG examples into the context. Each example contains a query, retrieved documents, and the expected answer. With a large context window, the system can pack in many such examples and ask the model to infer how to find the right answer from different document sets. The relevant settings include the number of retrieved documents, the number of examples, and the number of retrieval and generation iterations. ### Task-specific informativeness explains why the same setting works differently across datasets [04:54](https://www.youtube.com/watch?v=eEvramjyy-w&t=294s) The proposed equation includes a vector that measures the informativeness of each additional document and in-context example. This value is learned for each dataset and task because a document collection may contain either useful or weak evidence. Moving from zero-shot to one-shot, or from retrieving no documents to retrieving some documents, can therefore have very different effects depending on the task. The model also uses a logarithmic relationship because performance gains tend to diminish as more documents and examples are added. ### The model uses an inverse sigmoid to describe nonlinear performance [06:42](https://www.youtube.com/watch?v=eEvramjyy-w&t=402s) The speaker explains that performance is difficult to model directly because it is nonlinear and has different regimes. The model therefore includes an inverse sigmoid along with the logarithmic terms. Its learned coefficients are compared with measured results across configurations such as zero, one, ten, and one hundred documents. The predictions are not equally accurate for every dataset, but they generally remain within the observed bounds. ### Generalization to unseen tasks is the part the speaker finds least settled [09:05](https://www.youtube.com/watch?v=eEvramjyy-w&t=545s) The paper tests whether a model learned from known tasks can predict performance for tasks or datasets it has not seen. The speaker is unconvinced that the evidence fully establishes generalization to unseen production data. A further complication is that the system still needs to derive the task-specific informativeness vector at inference time for the target domain. The talk leaves open how teams would operationalize that derivation. ### Performance extrapolation changes across context-length regimes [11:40](https://www.youtube.com/watch?v=eEvramjyy-w&t=700s) The approach can sometimes extrapolate performance from a shorter effective context to a longer one, but the accuracy depends on the transition. The speaker gives 16k to 32k as a relatively successful example, while 32k to 128k is difficult. Extrapolation from 128k to one million is described as reasonably close, whereas one million to five million is poor. The speaker connects this variation to changes in which retrieval method performs best at different context lengths. ### More retrieval improves recall without guaranteeing better relevance [13:22](https://www.youtube.com/watch?v=eEvramjyy-w&t=802s) Including more documents generally raises recall, eventually approaching full coverage, but relevance and ranking quality do not improve at the same rate. Extra documents can introduce substantial noise. The speaker says this creates a need for better retrieval methods and for checks that the retrieved material is accurate and current. The system also needs to identify which documents matter most when many similar documents appear in the context. ### Errors come from retrieval, reasoning, evaluation, and refusals [14:04](https://www.youtube.com/watch?v=eEvramjyy-w&t=844s) The discussion separates errors into inaccurate or outdated retrieval, incorrect reasoning or missing reasoning, hallucinated or unfaithful reasoning, evaluation problems, and refusals to answer. Complex questions that depend on several supporting facts place particular pressure on retrieval and on the quality of the underlying knowledge base. Document ordering is another possible source of variation, since the ranking and placement of retrieved material may affect which evidence the model uses. ## Notable quotes - "We want to find the ones that produce the highest performance and to try to model the relationship between the hyper parameters and that performance." (01:13) - "The model's ability to identify relevant information from extensive context remains to be improved, especially when presented with large quantity of similar documents." (15:43) - "The more documents you include, you are increasing your recall." (13:22) - "We have to be investing in much better methods of retrieval." (13:43) ## Tools & references mentioned - RAG - IterDRAG - Sophia - Woodcraft ## Who should watch - You are tuning a RAG system and need to decide whether more retrieved documents or more in-context examples are worth the context cost. - You want to understand how task-specific document quality affects scaling experiments across datasets. - You are evaluating long-context behavior and need a sober account of where extrapolation and generalization can fail. ## Related talks - [Supercharging Your RAG System: Techniques and Challenges](https://mlopstalks.com/talks/supercharging-your-rag-system-techniques-and-challenges) (Tengyu Ma, Voyage AI and Stanford University, 40:20) - [The Future of AI: Long-Context RAG](https://mlopstalks.com/talks/the-future-of-ai-long-context-rag) (Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Sophia Skowronski, Breckinridge Capital Advisors & Adam Becker & Binoy Perera, MLOps Community, 49:19) - [Exploring Long Context Language Models](https://mlopstalks.com/talks/exploring-long-context-language-models) (Nehil Jain, Stealth AI Startup & Sonam Gupta, AICamp & Korri Jones, Chick-fil-A Corporate Support Center & Binoy Perera, MLOps Community, 49:25) - [Building Advanced Agents Over Complex Data](https://mlopstalks.com/talks/building-advanced-agents-over-complex-data) (Jerry Liu, LlamaIndex, 16:49) - [A Deep Discussion with the Author of "Context Rot"](https://mlopstalks.com/talks/a-deep-discussion-with-the-author-of-context-rot) (Kelly Hong, Chroma & Bauke Brenninkmeijer, orq.ai & Matt Squire, Fuzzy Labs & Arthur Coleman, Online Matters, 1:00:29)