Naive RAG is easy to prototype, but performance degrades as questions become more complex and data sources multiply.
2
Production RAG requires tuning choices across parsing, chunking, embeddings, retrieval, reranking, prompts, and output formatting.
3
More capable models will allow RAG systems to use repeated reasoning, search across complex problems, and take actions.
Summary
Jerry Liu explains why a five-line RAG prototype becomes difficult to operate in production. Simple retrieval works for a fact in one document, but quality drops when users ask multi-part questions or the system must combine information across many files. The pipeline has interacting choices at every stage, including document parsing, chunking, metadata, embeddings, top-k retrieval, reranking, prompts, and output formats. Liu groups failures around missing data, failed retrieval, poor extraction, incorrect or incomplete answers, scalability, security, and difficult document types. He recommends improving parsing and metadata, tuning retrieval settings, using larger initial retrieval sets with reranking, considering task-specific embedding fine-tuning, compressing or reordering context, and enforcing structured output. Complex PDFs need a document hierarchy that preserves tables and spatial relationships. Liu expects future systems to add agentic reasoning, query planning, repeated retrieval, and tool use.
Naive RAG works for simple facts, then breaks as the problem grows
Liu says naive RAG is easy to prototype and works well when a user asks for a specific fact in a small, simple document set. His examples include finding risk factors in a Tesla 10-K or asking what Paul Graham did during his time at Y Combinator. Quality falls when questions have multiple parts, require evidence from separate documents, or span many books and PDFs. The symptoms include bad retrieval and hallucinations. Teams also struggle to identify which part of the system needs improvement because many parameters affect the final answer.
Every RAG component becomes part of one hard-to-reason-about system
Traditional software has programmatic rules that make expected outputs relatively easy to reason about. An AI application contains black-box model behavior, and the surrounding choices become part of the same difficult function. Liu uses chunk size as an example. Splitting text into 256-token or 512-token chunks is easy to test locally, but its effect on end-to-end answer quality cannot be known without evaluating the whole application on a dataset. Developers therefore face a combinatorial set of interacting decisions.
Production teams must tune the full retrieval pipeline
For PDFs, Liu lists choices such as the parser, chunking strategy, chunk size, treatment of tables and charts, embedding model, retrieval method, top-k value, hybrid search, and prompt template. He proposes organizing these decisions by failure category, locating each failure in the indexing or query pipeline, and then applying practices suited to that stage. He cites work that maps failures from document processing through query rewriting, retrieval, reranking, and synthesis.
Better source data and metadata improve what retrieval can find
Sometimes the requested context is absent from the knowledge base, or it was damaged during parsing. Liu says parser choice matters, and reports that LlamaIndex has more than 10 PDF parser integrations with widely varying results. Even details such as retaining or removing HTML tags can change performance. Adding metadata to each chunk gives the embedding model, vector database, and language model information about the source article and the chunk's relevance. Production systems also need recurring ingestion because their sources change over time.
Retrieval needs debugging before more sophisticated techniques
When relevant context exists but is not retrieved, Liu recommends tuning chunk size and top-k. A simple diagnostic is to raise top-k substantially and check whether the desired context appears at all. If it still does not appear, the data representation may be the problem. If it appears in the larger result set, the retrieval algorithm or embedding model needs work. Teams can retrieve a large candidate set first and then apply reranking or filtering. They can also use hybrid search, query planning, and different chunk representations.
A model can receive the right context and still miss the answer
Liu describes a separate failure in which retrieval succeeds but the language model cannot extract the needed information. He connects this to needle-in-a-haystack tests, where a fact placed in the middle of a long context can be missed. Prompt compression can reduce token cost and latency while retaining useful information. Another experiment is context reordering, placing the most relevant chunks at the beginning and end so less relevant material sits in the middle. Liu presents this as a practical trick whose results can vary.
Many RAG applications need JSON rather than free-form text. Liu describes three approaches: instructing the model through text prompts, using OpenAI function calling, and enforcing token patterns during decoding with tools such as Guidance and LMQL. Token-level methods can constrain generation directly when developers have access to the model. LlamaIndex integrates with Guidance, which Liu recommends trying for applications that need consistent structured output.
Complex questions need an agentic layer above basic RAG
Top-k retrieval is mainly suited to simple factual questions. A multi-part question may require several searches and intermediate steps before the system can answer. Liu describes adding query planning, execution loops, tool use, and a ReAct-style process on top of a RAG pipeline. This approach can break a complex question into smaller tasks and support longer research problems. The agent then combines information from the RAG system with other sources.
Tables require document structure that preserves their meaning
Naive parsing often splits tables, collapses their contents, and destroys spatial relationships. Liu says this can cause hallucinations when the resulting text is indexed. His proposed approach models a PDF as a hierarchy of nodes. Text is split by section, while tables are represented through extracted content and summaries. These nodes can be linked into a document graph and indexed for semantic retrieval. He also previews a LlamaIndex parser for complex PDFs that extracts tables and supports recursive retrieval over semi-structured data.
