RAG remains necessary for proprietary, unstructured company data, even as language models become more capable.
2
Embedding models retrieve a broad set of candidates, while rerankers compare each query-document pair more carefully and select a smaller final set.
3
Retrieval quality depends on the data, chunking, query strategy, models, and evaluation set, so teams need experiments rather than a universally reliable technique.
Summary
Tengyu Ma explains why retrieval-augmented generation remains the practical way to give language models access to proprietary company knowledge. He compares RAG with fine-tuning and long-context prompting, arguing that fine-tuning needs substantial data and long context is expensive. He then walks through the RAG pipeline, from extracting and transforming documents to embedding, vector search, reranking, and generation. Embedding models are fast because document vectors can be precomputed, while rerankers are more accurate because they see the query and document together. Ma discusses query expansion, multiple queries, iterative retrieval, chunk sizing, document titles, domain-specific models, fine-tuning, and evaluation. He is direct about the need to test each choice on a representative dataset. Specialized domains and missing global document context remain open problems, and he wants the field to reach consistently high retrieval accuracy in production.
RAG adds company knowledge that a general language model does not have
Ma argues that a general language model should not be expected to know a company's institutional knowledge or proprietary data. RAG adds a knowledge layer on top of the model by retrieving relevant material from the company's unstructured data. The pipeline extracts and transforms source data into text, creates embeddings, stores vectors, searches for candidate documents, reranks them, and gives the best results to the language model. Ma says the extraction and vector database stages are mostly engineering concerns, while embedding models and rerankers determine retrieval quality. If the retrieved information is relevant, even a relatively weak language model can often produce a good answer.
RAG is cheaper and more adaptable than fine-tuning or reading a huge context
Ma compares RAG with fine-tuning and long-context models using a biological analogy. RAG retrieves books from a library, fine-tuning rewires the brain, and long context reads the entire library into short-term memory. Fine-tuning can work well when enough data is available, but small datasets can cause weak knowledge transfer or overfitting. Long context avoids retrieval, but reading a very large amount of text for every query is expensive. In his rough example, a query over one million tokens has a much higher cost than RAG because RAG depends mainly on the query length. He expects both approaches to become cheaper, but says the same engineering improvements can also reduce RAG costs.
Embedding search is fast because documents can be processed before the query arrives
An embedding model maps documents and queries into vectors. Semantic relatedness is approximated by geometric distance, so search finds document vectors that are close to the query vector. The important engineering advantage is that document embeddings can be computed and stored before query time. The limitation is that a document embedding is created without knowing the future query. It must capture many possible aspects of the document, which can make it difficult to preserve every detail. Vector search therefore works well as a broad pre-filter, but it may return documents that are only partly relevant.
Rerankers improve precision by reading the query and document together
A reranker takes a smaller set of candidate documents and assigns each one a relevance score. Unlike a bi-encoder embedding model, a cross-encoder sees the query and document together. It can focus on the parts of the document that matter for that specific query, which gives it a more accurate relevance judgment. The tradeoff is speed. The model must process each query-document pair at query time, so it cannot efficiently score thousands or millions of documents. Ma describes the normal combination as vector or lexical search first, followed by reranking of perhaps a much smaller candidate set before sending the best documents to the language model.
Query expansion helps retrieval when the original question is too short or ambiguous
Ma recommends expanding queries when the original wording gives the retrieval system too little information. A short query such as an acronym may be unclear to an embedding model or reranker. Adding an explanation and relevant terms can make matching easier. Teams can use rules or a language model to produce a more detailed query, or generate several queries that approach the problem from different angles. He also suggests stating the desired definition of similarity when it is not obvious, since a model may otherwise apply a generic meaning of similarity. Whether this instruction helps depends on how responsive the reranker is to such guidance.
Retrieval can be improved by following links and choosing chunk boundaries carefully
Some answers cannot be found in one retrieval step. Documents may link to other documents, or the first useful result may point toward the evidence needed next. Iterative or recursive retrieval can follow those connections with rules, embeddings, or a language model. Chunking creates another tradeoff. Smaller chunks allow more chunks to be retrieved within a downstream token budget, while larger chunks contain more context and may be easier to interpret. Ma says a chunk of 512 tokens can sometimes be too small, while the longest available context is not automatically best. The right choice depends on cost, model behavior, and evaluation.
Document structure and titles can determine whether a chunk remains understandable
Ma advises inspecting failed retrievals instead of applying techniques blindly. A chunk may be technically present but meaningless without the document title or surrounding context. In legal documents, for example, company names may appear early and later sections may refer only to roles such as licensor or licensee. A later chunk without the title or opening context may not identify the parties at all. Copying the document title or other useful headers into each chunk can make it interpretable to both models and people. He also recommends trying stronger general-purpose or domain-specific embedding and reranking models, then fine-tuning when a suitable set of positive query-document pairs is available.
RAG systems need evaluation because no retrieval technique works in every dataset
Ma says every retrieval change should be tested against a proper evaluation set before deployment. Teams can create labelled relevance data or use a language model as a judge to compare documents. When using a model judge, he recommends anonymizing model names and randomizing result order because a fixed order can introduce bias. Evaluation also determines chunk size, query expansion, recursive retrieval, and model selection for a particular dataset. New documents do not require retraining the whole RAG system. They need to be embedded and added to the vector database. Changing the embedding model does require re-embedding the document collection, while changing a reranker can usually be done by calling a different model.
"Even large language models approach AGI, even you have Einstein, you still need some retrieval system, some knowledge system that helps you ingest the proprietary and unstructured data from the proprietary environment."04:33
Who should watch
You are building a RAG pipeline and need to decide how vector search, reranking, chunking, and query expansion should fit together.
Your system retrieves documents that contain the answer, but the generated response still misses the relevant evidence.
You need a practical way to evaluate retrieval changes, update a document collection, or decide whether a new embedding model is worth reindexing for.
