The paper introduces the Loft benchmark to compare long-context language models with retrieval, RAG, and SQL systems across several tasks.
2
Long-context models can approach or exceed specialised retrieval systems on some text, visual, audio, and multi-hop tasks, but their performance depends heavily on prompt structure and document position.
3
The group does not think the results justify removing RAG or trusting the paper's model comparisons without more independent and reproducible evaluation.
Summary
Can Long-context Language Models Subsume Retrieval, RAG, SQL, and More? is the paper discussed in this reading group. Sonam Gupta introduces long-context language models and the paper's Loft benchmark, which tests retrieval, question answering, multimodal inputs, few-shot learning, and SQL-related tasks at different context lengths. The discussion focuses on how corpus-in-context prompts are formatted, including document IDs, few-shot examples, chain-of-thought reasoning, and placing the query at the end. Nehil Jain explains that prefix caching can reduce repeated computation when the same large corpus is used for several questions. The paper reports strong results for some retrieval and multi-hop RAG tasks, while SQL remains difficult. Participants question whether the results generalise beyond Gemini, whether the comparisons are fair, and whether loading an entire corpus is cheaper than retrieval. The group sees long-context models as useful for experiments and some workloads, but does not view the paper as evidence that teams should remove RAG from production systems.
Long-context models are designed to process whole documents and conversations
Sonam Gupta describes large-context language models as models with much larger context windows than earlier systems. They can analyse entire articles, books, legal documents, or long conversations instead of short snippets. She connects this capability to tasks such as summarising a lengthy book and working through complex legal material. The paper asks whether this extra context can replace systems built around retrieval-augmented generation or SQL queries. Sonam also points out the direct cost of the approach: putting an entire book or corpus into a model can require substantial computation and resources.
The Loft benchmark tests whether one model can absorb retrieval and structured-data workflows
The paper's authors build the Loft benchmark to evaluate long-context language models across different context sizes and task types. Sonam describes tests involving information retrieval from large document collections, question answering, multimodal data, and natural-language requests that normally require SQL. The evaluation uses shared queries across context lengths, with datasets drawn from retrieval, conversational question answering, BigBench Hard, LongBench, and SQL benchmarks including Spider and SPARC. The group discusses Gemini 1.5 Pro, GPT-4, and Claude 3 as evaluated models. The benchmark is intended to measure what a model can do when the relevant corpus is placed directly in context rather than retrieved externally.
Corpus-in-context prompting depends on careful formatting
Sonam explains the paper's corpus-in-context prompting method. The prompt has task instructions, formatted corpus content, few-shot examples, and a formatted query. Retrieval passages receive document IDs at their beginning and end, while examples show the model what the question and answer format should look like. The query is formatted to match those examples. The group also discusses chain-of-thought reasoning, especially for multi-hop question answering. Later, Nehil says that assigning IDs to passages and repeating those IDs in the output improves retrieval and reasoning. The discussion treats prompt design as part of the system being evaluated, rather than as a minor implementation detail.
Long context still has a positional weakness and performs best when the query comes last
The participants connect the paper to the lost-in-the-middle problem. A model may fail to attend to information placed in the middle of a very large prompt, even when its context window is large enough to hold the material. Nehil describes the paper's positional analysis and says that performance is better when the corpus comes first and the query comes at the end. Prefix caching can then encode the shared corpus once while later requests change only the examples and query. This reduces repeated computation, although it does not remove the cost of processing the large context in the first place.
Long-context models can match specialised retrieval systems on some tasks
For text retrieval, the paper compares long-context models with specialised embedding systems, including Google's Gecko model. Nehil reports that the long-context approach is nearly comparable to, or better than, the specialised retrieval approach in the reported tests. The group also discusses visual retrieval over frames and audio inputs, where the model comparisons depend on which systems accept those modalities. The participants are careful about the result because the prompts were optimised for Gemini and then used with other models. They also note that placing relevant few-shot examples near the material to be retrieved may improve results, though this makes the evaluation setup feel less neutral.
Long context helps multi-hop RAG more clearly than multi-target RAG
Nehil distinguishes multi-hop and multi-target tasks. Multi-hop questions require the model to find and combine information from several sources. Multi-target questions ask for several answers from one document. The paper reports a stronger advantage for long-context models on multi-hop tasks, especially when chain-of-thought reasoning is used across the whole corpus. The group discusses how this differs from iterative RAG, where a system retrieves, reasons, and then retrieves again based on the intermediate result. The paper's approach places the corpus in context and reasons over it in one setup, which may reduce orchestration complexity while also making the computation larger.
Long-context models do not solve text-to-SQL in the reported evaluation
The discussion treats SQL as a remaining weakness. Nehil explains that the benchmark asks models to turn difficult business questions into SQL over databases. A specialised system such as DAIL-SQL decomposes a question into parts, generates SQL for those parts, and combines the results. The paper reports that long-context models still perform poorly on SQL reasoning compared with the specialised approach. The group mentions that some SQL operations, such as equality conditions, appear easier than aggregate operations such as averages and sums. The result does not support replacing database querying with a long prompt.
The paper's benchmark is more convincing to the group than its broad model claims
The participants separate the paper's evidence from its conclusions. One participant says the benchmark itself feels useful and defensible, while the ablation study and claims about Gemini appear more biased. Korri Jones asks whether the results can be repeated and whether the paper gives a complete picture or an idealised one. Sonam says the work points in a useful direction but should be treated cautiously. The group also notes that the field still lacks settled datasets, standards, and measures for evaluating language models. The practical conclusion is to use the paper to guide experiments, not to remove existing retrieval systems based on one benchmark.
"The title of the paper is Can Long-context Language Models Subsume Retrieval, RAG, SQL and More?"Sonam Gupta01:39
Who should watch
You are deciding whether a long-context model can replace a retrieval or RAG pipeline and want to understand the trade-offs discussed by practitioners.
You need to evaluate long-context models and want examples of how corpus formatting, query position, few-shot examples, and document IDs affect results.
You read model benchmarks but want a discussion of reproducibility, model-specific prompt optimisation, and the limits of broad claims about production use.
