Naive RAG works for simple questions over a small number of simple documents, but it breaks down on complex files, large document collections, and multi-part questions.
2
Parsing quality affects answer quality directly, especially when documents contain tables, charts, images, spatial layouts, or other non-text elements.
3
A stronger pipeline parses documents into different element types, indexes multiple text representations, retrieves those representations, and then fetches the original source elements.
Summary
Jerry Liu argues that data quality is a necessary part of production LLM applications. Basic RAG is easy to prototype with document splitting, dense retrieval, and prompt stuffing, but this approach fails on complex documents and broader question sets. Tables, charts, images, and page layouts can be damaged by naive parsing, which leads to incorrect answers even when using powerful models. Liu describes a pipeline that preserves document structure, creates representations for text, tables, and images, and links those representations back to their source elements. He also recommends page-level chunking as a strong baseline, preserving semantically related content, extracting metadata, and using a document store for source files, caching, and incremental updates. Conversation history and longer-term memory also need storage. He expects retrieval to remain useful as context windows grow, while fine-grained chunking may become less important. He also sees a need for storage systems that combine vector, SQL, and knowledge-graph queries across structured and multimodal data.
Naive RAG is useful for simple questions over simple documents
Liu describes the common first RAG pipeline: parse documents with an open-source parser, split them into chunks, retrieve the top results with dense embeddings, and place them in the prompt. This can be built in roughly five to ten minutes by someone comfortable with Python. It works reasonably well when a user asks about a specific fact in one of a few PDFs. Embeddings can surface the relevant chunk, and an LLM can usually produce an answer from that context. The approach becomes much less dependable as the questions, document count, or document structure grow.
Complex data creates failures even when the question is simple
The difficult cases include simple questions over complicated files, questions that span many documents, and vague or multi-part questions. Complex files can contain embedded tables, PowerPoint layouts, SVG shapes, rasterized images, headers, and footers. Naive indexing may fail to preserve the relationships among these elements. A user asking about a table, chart, or image can receive a hallucinated answer because the relevant content was never represented correctly before retrieval.
Parsing quality directly changes the answers an LLM can produce
Liu uses a financial-report table to show how a poor PDF parser can blend numbers and text into a messy sequence. When the table loses its structure, models such as GPT-4o and Opus may hallucinate a value in response to a question about the table. LlamaIndex's LlamaParse is designed to extract and format tables and charts, accept natural-language parsing instructions, extract images for multimodal RAG, and handle PDFs, PowerPoints, DOCX files, and HTML. Liu says that better formatting alone can reduce hallucinations, even before adding more advanced retrieval.
Hierarchical retrieval links searchable representations back to source elements
A complex document can be parsed into text chunks, tables, images, and other multimodal elements. The system then creates one or more text representations for each element, such as a table summary, image description, or table-cell text. Those representations are embedded and indexed, while links preserve access to the original table, image, or text. During recursive retrieval, the system first finds the indexed summaries or sentences, then follows the links to fetch the source elements. This gives one document element several searchable representations and lets the final answer use the original data.
Page-level chunks are a practical baseline for many documents
Liu recommends page-level chunking for many PDF and PowerPoint collections. A page often contains much of the information needed to answer a question, so this approach avoids spending too much time tuning an exact chunk size. It has exceptions, such as sections that span multiple pages, where metadata and filters may be needed. He also recommends keeping semantically related content together, rather than cutting through a table or the middle of a section. If long-context models become cheaper and faster, document-level chunking may reduce the need for fine-grained chunk-size decisions.
Metadata and multiple representations make retrieval more flexible
Metadata gives documents a semi-structured form that can be queried across different dimensions, including vector search and SQL. Liu says a single vector for one text chunk is often insufficient. A system may create a summary, a sentence, or other representations of the same underlying object and index each one. Retrieval can then find the source object through whichever representation matches the question, deduplicate results, and return the underlying data rather than only the retrieved text fragment.
A RAG system needs storage beyond a vector database
Liu recommends a document store or key-value store alongside a vector database. It can hold source documents and hierarchical information, support caching, and enable incremental syncing when only some documents have changed. Storing document hashes allows the system to identify which files need updating. Agents and chatbots also need a place for conversation history and longer-term memory. Liu expects future systems to retrieve these memories in structured forms, potentially with knowledge graphs, alongside document content.
Longer context will change chunking, while retrieval will remain useful
Liu expects retrieval to remain useful for large document collections even as long-context models improve. He thinks fine-grained decisions about chunk size within a page may become less important as context windows get cheaper and faster. He also points toward a unified storage system that can search with vectors, SQL, and knowledge-graph queries across structured, unstructured, and multimodal data. As models become more capable with images, an image of a page could become a native document representation that preserves diagrams and other visual information.
