# The Future of AI: Long-Context RAG

Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Sophia Skowronski, Breckinridge Capital Advisors & Adam Becker & Binoy Perera, MLOps Community | MLOps Reading Group | 49:19
Hosted by Adam Becker

Source: https://www.youtube.com/watch?v=YK18P52ab-k
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-future-of-ai-long-context-rag
Published: 2024-12-02
Tags: inference, long-context, rag

## TL;DR
- The paper studies how long-context models can improve RAG by spending more inference-time compute on documents, examples, and iterative query decomposition.
- Iterative demonstration-based RAG helps with questions that require several retrieval steps, such as linking an Academy Award-winning film to its author.
- The paper's scaling model can predict useful parameter settings in some context-length ranges, but the discussion raises doubts about generalization to unseen tasks and domains.

## Summary
The paper "Inference Scaling for Long-Context Retrieval Augmented Generation" studies how to allocate long-context inference compute for retrieval-augmented generation. Valdimar Eggertsson explains that the authors combine large context windows with in-context examples, then compare ordinary RAG, demonstration-based RAG, and iterative demonstration-based RAG. Iterative RAG decomposes a complex question into subqueries, retrieves documents for each step, and combines intermediate answers. Sophia Skowronski walks through the paper's search over documents, demonstrations, and iterations under a fixed context budget. Adam Becker explains the proposed model for predicting performance from those parameters and discusses its validation. The group notes that more retrieved documents can improve recall while also adding noise, and that gains can plateau or decline when the context becomes too large. The discussion is candid about weak evidence for generalization to unseen datasets, limited comparisons with other models, and uncertainty about how to operationalize the paper's task-specific informativeness estimates.

## Key ideas
### Long-context models let RAG spend more inference compute on a question
[01:00](https://www.youtube.com/watch?v=YK18P52ab-k&t=60s)
The paper studies inference scaling by putting more content into the RAG process. Valdimar Eggertsson contrasts ordinary retrieval setups, which may use around 20,000 tokens, with long-context models that can accept over a million tokens. This makes it possible to place much more retrieved material and many task examples in the prompt, although the added context costs compute. The approach combines in-context learning, where the model sees demonstrations before answering, with iterative prompting, where the model repeats a retrieval and generation process. The paper asks how performance changes as this inference budget grows and whether the best allocation can be predicted for a given budget.

### A large context window does not guarantee that the model will find the relevant fact
[08:23](https://www.youtube.com/watch?v=YK18P52ab-k&t=503s)
Valdimar Eggertsson says current long-context models still struggle to locate relevant information in ultra-long sequences. Simply placing an entire database or book in the prompt can bury useful facts in noise. The paper therefore treats retrieval as useful even when context windows are very large. The group discusses how long-context systems can process sequences of millions of tokens despite the quadratic memory cost associated with standard attention, mentioning sparse, low-rank, and efficient attention methods. The practical issue remains information selection: more available context gives the model more material, but it does not ensure that the model will identify the right passage.

### Demonstration-based RAG teaches the model how to use retrieved documents
[09:13](https://www.youtube.com/watch?v=YK18P52ab-k&t=553s)
Demonstration-based RAG, called DRAG in the discussion, adds examples of successful retrieval and answering to the prompt. Each example contains retrieved documents, a question, and an answer that shows how the relevant information can be found and used. The demonstrations give the model a pattern for working with the current documents. Valdimar Eggertsson says this produces a performance increase over ordinary RAG in the paper's experiments. The approach uses the long context window to include many examples, rather than relying only on a single query and a single retrieval result. It is especially useful when the model needs guidance about how to filter documents and construct an answer.

### Iterative DRAG improves questions that require several retrieval hops
[10:03](https://www.youtube.com/watch?v=YK18P52ab-k&t=603s)
Iterative DRAG extends the demonstration approach with query decomposition. Instead of searching once for the full question, the model breaks it into simpler subqueries, retrieves documents for each one, produces an intermediate answer, and uses that answer in the next step. The example discussed is a question that first requires identifying the film that won the 2003 Academy Award for Best Film and then finding who wrote that film. The group reports that performance on a multi-step question-answering set rises from 50% with ordinary RAG to 76.9% with iterative decomposition. The method can repeat this process several times, using a large context for demonstrations and intermediate results.

### The best allocation depends on the context budget and the retrieval strategy
[17:45](https://www.youtube.com/watch?v=YK18P52ab-k&t=1065s)
Sophia Skowronski describes the paper's parameter search under a maximum context length, L Max. The variables include the number of retrieved documents, the number of in-context examples, and the number of iterations. The experiments compare iterative DRAG, DRAG, zero-shot question answering, many-shot question answering, and basic RAG. DRAG works better in smaller long-context windows, while iterative DRAG becomes more effective at 128K tokens and above. The results do not support one universal setting. The useful choice depends on how much context is available and on whether the task benefits from repeated decomposition. Some methods also stop scaling because their own structure cannot use the full context budget.

### More documents help until extra context becomes noise
[26:13](https://www.youtube.com/watch?v=YK18P52ab-k&t=1573s)
The presenters describe performance as rising with effective context length, with diminishing returns. Increasing the number of retrieved documents generally helps, but the benefit eventually plateaus and can reverse for some settings. Increasing the number of shots is often more useful for iterative DRAG than for DRAG, while a single additional example can produce a large improvement in some ranges. The discussion separates recall from relevance and ranking quality. Retrieving more documents can increase recall without improving the order or usefulness of those documents, and similar documents can make it harder for the model to identify the right evidence. Retrieval quality therefore remains a central limit even when the model can accept a very large prompt.

### The paper models performance with task-specific informativeness
[33:09](https://www.youtube.com/watch?v=YK18P52ab-k&t=1989s)
Adam Becker explains the paper's attempt to predict the best configuration instead of testing every possible combination. Its model relates performance to the inference parameters, including documents, demonstrations, and iterations. It also includes a task-specific vector that measures how informative additional documents or examples are for a dataset and task. The model uses logarithmic relationships to capture diminishing returns and an inverse-sigmoid form for nonlinear performance. Adam says this task-specific component is important because the same extra document or example can have different value on different tasks. The group remains uncertain about how a production system would estimate this informativeness for a new target domain.

### Generalization across context ranges and unseen tasks remains uncertain
[38:00](https://www.youtube.com/watch?v=YK18P52ab-k&t=2280s)
Adam Becker is skeptical about how well the scaling model generalizes beyond the tasks used to fit it. The paper can sometimes extrapolate from lower to higher effective context lengths, but the results are weaker when moving across regimes where DRAG and iterative DRAG have different strengths. The discussion gives 32K to 128K as a difficult transition, while some other context-length transitions are easier to predict. Adam also questions whether the reported validation really demonstrates generalization to unseen datasets, since the task-specific informativeness estimate still has to be derived for the target domain. Participants raise concerns about parameter selection on test results, the use of Gemini 1.5 Flash alone, and the limited comparison with other models and approaches.

## Notable quotes
- Valdimar Eggertsson: "It doesn't work that well to just put all the info there because it gets lost in the noise." (08:23)
- Adam Becker: "The more documents you include, you are increasing your recall, but it doesn't stop. At some point you're going to get to 100%, but all of the other metrics don't seem to be increasing all that much and you might be introducing noise." (42:07)
- Adam Becker: "The fact that you have to still have in the inference time to derive I from the target domain makes it difficult to operationalize." (39:44)
- Sophia Skowronski: "Increasing the context no longer helps or appears to be confusing the model in this range." (27:37)

## Tools & references mentioned
- Inference Scaling for Long-Context Retrieval Augmented Generation
- Google DeepMind
- GPT-3
- GPT-3.5
- GPT-4
- Gemini 1.5 Flash
- Chain-of-Thought
- Wikipedia
- Academy Awards
- Return of the King

## Who should watch
- You are designing a RAG system and need to decide whether to spend more inference compute on retrieved documents, examples, or repeated retrieval steps.
- You work with multi-hop questions where one search result supplies the query for the next search.
- You want a careful discussion of long-context scaling claims, including task generalization, retrieval noise, and evaluation design.

## Related talks

- [Exploring Long Context Language Models](https://mlopstalks.com/talks/exploring-long-context-language-models) (Nehil Jain, Stealth AI Startup & Sonam Gupta, AICamp & Korri Jones, Chick-fil-A Corporate Support Center & Binoy Perera, MLOps Community, 49:25)
- [RAG Has Been Oversimplified](https://mlopstalks.com/talks/rag-has-been-oversimplified) (Yujian Tang, Zilliz, 48:56)
- [A Deep Discussion with the Author of "Context Rot"](https://mlopstalks.com/talks/a-deep-discussion-with-the-author-of-context-rot) (Kelly Hong, Chroma & Bauke Brenninkmeijer, orq.ai & Matt Squire, Fuzzy Labs & Arthur Coleman, Online Matters, 1:00:29)
- [RagSys: RAG is just RecSys in Disguise](https://mlopstalks.com/talks/ragsys-rag-is-just-recsys-in-disguise) (Chang She, LanceDB, 12:45)
- [RAG: The Future of AI Search (and Prompt Engineering)](https://mlopstalks.com/talks/rag-the-future-of-ai-search-and-prompt-engineering) (Korri Jones, Chick-fil-A Corporate Support Center & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Sophia Skowronski, Breckinridge Capital Advisors & Lihu Chen & Binoy Perera, 10:13)