Longer inputs reduce LLM performance, even on the relatively simple needle-in-a-haystack retrieval task.
2
Needle-in-a-haystack benchmarks can overstate long-context ability because they often use lexical matches and do not test whether a model understands the rest of the context.
3
Focused context generally performs better than full context, while distractors and longer inputs make retrieval harder.
Summary
Kelly Hong presents Context Rot, a Chroma technical report about how LLM performance changes as input length grows. She argues that million-token context windows do not mean models can use every token equally well. Needle-in-a-haystack tests are easy to scale, but they often rely on simple retrieval and word overlap, so they do not capture coding agents, summarization, or other tasks that require attention across a whole context. The discussion covers experiments that vary needle-question similarity, needle-haystack similarity, distractors, and document structure. LongMemEval results show that full chat histories often reduce performance compared with focused inputs, especially for some smaller or non-thinking models. Reasoning modes can improve results on complex questions. The speakers also discuss shuffled text, error bars, reproducibility at temperature zero, and whether filtering context is more useful than rearranging it. Adam Becker closes by asking how ambiguity should be measured from the user's side, rather than only through similarity between a question and a retrieved passage.
Million-token context windows do not guarantee uniform model performance
Kelly Hong says Gemini, GPT-4.1, and Sonnet 4 advertise context windows of up to one million tokens, but that presentation can imply that a task at 1,000 tokens will work as well as the same task at one million tokens. The report tests that assumption. Hong describes context rot as the lower-quality output people notice during long ChatGPT conversations or when using coding agents such as Claude Code. The research measures performance across the context window instead of treating the advertised maximum as the effective working length.
Needle-in-a-haystack tests are easy to scale but too simple on their own
The standard test places a fact, the needle, inside a long collection of documents, the haystack, then asks the model to retrieve it. Hong says this is useful because input length and needle position are easy to control. It also has limits. The task asks the model to find one fact and often uses a question and answer with substantial word overlap. That can reward lexical matching without showing whether the model processed the surrounding content. A coding agent with code files, tools, and conversation history requires much more than this single-fact retrieval task.
Question and needle similarity changes how quickly performance falls
Adam Becker explains the experiment that varies the similarity between the question and the hidden answer. The researchers create needles that range from highly similar to the question to more semantically related but lexically different. They embed source text, reduce its dimensions, cluster it, review representative passages, and create questions and answers that fit the source material. The results show degradation as input length grows. Highly similar question-needle pairs hold up longer, while less similar pairs decline faster. Becker says the effect appears across high-, medium-, and low-performing model groups.
LongMemEval shows that irrelevant context can reduce complex question-answering performance
Bauke Brenninkmeijer discusses LongMemEval, which evaluates questions over long chat histories involving knowledge updates, temporal reasoning, and multi-session conversations. The experiments compare the full input with a focused input containing only passages relevant to the question. Full-context performance is generally lower, with larger drops for some smaller or older models. The discussion also compares thinking and non-thinking modes. For Claude, thinking produces a large improvement on temporal reasoning, multi-session questions, and knowledge updates. Brenninkmeijer presents this as evidence that long-context performance depends on the model and the type of reasoning required.
The speakers disagree about how much evidence temperature-zero results provide
When asked about missing error bars, Kelly Hong says the models were run at temperature zero and that each result represented many binary question-answer outcomes. Samantha challenges the assumption that temperature zero guarantees identical outputs, based on her own experiments. Hong agrees that repeated runs and error bars would make the results more robust, while Adam Becker argues that distributions can reveal uneven failures across subsets of tasks. The exchange matters because a single accuracy average can hide whether a model fails rarely across many tasks or fails repeatedly on a particular class of questions.
A distractor can be related to the answer without answering the question
Becker describes distractors as content that is topically related to the needle but does not answer the question. For a question about the best writing advice from a college classmate, examples include advice from a professor, the worst advice from a classmate, or a different piece of advice from the classmate. The experiments add one or more distractors and measure retrieval as context grows. Performance gets worse as the number of distractors increases. Becker says one type of distractor appears especially harmful and received a separate error analysis.
Filtering relevant context is more useful than shuffling it
Matt Squire discusses an experiment that changes the relationship between the hidden needle and its surrounding haystack. Retrieval can improve when the needle is unrelated to nearby text, and shuffling sentences can also improve retrieval. The finding is surprising because a coherent essay is easier for a person to follow. Hong cautions against treating the shuffling result as a production technique. She recommends reducing irrelevant context and using retrieval or reranking to focus the model on material related to the question. She says filtering performed better than shuffling in their practical observations.
Ambiguity can be studied from the user's language as well as the model's retrieval task
Adam Becker says the paper's use of ambiguity made him want to study how people phrase questions in long conversations. He asked ChatGPT to identify similar questions, explain why they were ambiguous, and assign an ambiguity score. In his informal analysis, question length had little relationship with the score. He suggests that each person's way of speaking to a model may be a stronger factor than token count alone. Hong explains that the paper uses cosine similarity between question and needle embeddings as its quantitative definition of similarity, rather than trying to assign ambiguity to a single question.
"The main takeaway from that is to limit the amount of irrelevant context, limit the amount of distractors, just try to focus your context on as much relevant information as possible."Kelly Hong51:16
Who should watch
You are building a long-context application and need to know whether a larger context window will improve the actual task you care about.
You rely on needle-in-a-haystack or retrieval benchmarks and want to understand what those tests leave out.
You are designing evaluations for chat histories, coding agents, or document question-answering and need to account for distractors, similarity, and repeated-run variation.
