RAG systems are among the simpler LLM applications to put into production, either as complete systems or as components inside larger applications.
2
Production RAG requires decisions about chunking, embeddings, vector databases, retrieval, reranking, evaluation, cost, and latency.
3
LLM applications are harder to test than traditional software because prompts affect behavior, outputs are not deterministic, and a fix can worsen another part of the system.
Summary
Apurva Misra explains how companies are putting LLM applications into production, with RAG as the most common starting point. She walks through the pipeline from source documents to chunking, embeddings, vector storage, retrieval, reranking, and answer generation. Each stage has application-specific choices. These include chunk size, embedding model, database setup, hybrid search, filtering, and terminology that the model may not know. Misra then covers component-level and end-to-end evaluation, including the risks of using one LLM to judge another. Production systems also need to control cost and latency through model routing and caching. Her closing point concerns software development practice. LLM applications have opaque execution paths, prompt-sensitive behavior, nondeterministic tests, and token limits. A code change or prompt change can improve one case while making the overall system worse, so machine learning and software engineering teams need to communicate closely.
Misra says companies generally put RAG systems into production because they appear simpler to build initially. A system can use Confluence documentation, YouTube transcripts, help articles, or another source. The documents are split into chunks, embedded, and stored in a vector database. When a user asks a question, the system retrieves similar chunks and sends them to an LLM with the query. RAG can also sit inside a larger application. For example, a system might route a help-article question to RAG, another question to an API, and a different request to a search engine.
The extraction stage requires decisions about chunk size and the number of chunks. A PDF might be split one page at a time or into groups of pages. Some systems retrieve nearby chunks alongside the chunk that matches the query. Others create several representations of the same document, such as a page embedding and an embedding of a generated summary. Industry-specific terminology also matters when the embedding model does not know the terms. Misra advises testing several embedding models rather than assuming the highest-ranked model on the Hugging Face leaderboard will work best.
Misra says a simple PostgreSQL setup with PGVector can work well, while dedicated vector databases offer more functionality. The choice depends on factors such as on-premises versus cloud deployment, indexing speed, query latency, recall, and low-latency requirements. Teams may also need dense or sparse vectors, hybrid search, in-memory or on-disk storage, pre-filtering or post-filtering, and reranking. Hybrid search combines keyword matching with semantic similarity, which can help when important industry terms are missing from the embedding model's training.
Retrieval can combine semantic, keyword, and deterministic matching
After embeddings are stored, the system retrieves chunks that may answer the query. PGVector provides distance measures such as cosine distance and inner product. Misra says the top results may need reranking to determine which retrieved chunks are most relevant, and she mentions Cohere as a provider of a reranking API. Retrieval can also combine semantic similarity with keyword matching. A deterministic condition can narrow the result further, such as requiring documents to come from January through March 2023.
Evaluation must separate retrieval from answer quality
Misra divides RAG evaluation into component-level and end-to-end evaluation. At the component level, teams can test whether a query retrieves context from the vector database that is relevant to answering it. They can then test the quality of the answer given the query and correct context. She has seen companies ask an LLM to score a RAG system from one to five, but warns that the evaluator can be biased. For example, GPT-4 may rate GPT-4 answers as the best, so teams need to account for that bias.
LLM systems become more expensive as usage grows, so Misra says production teams need both quality and cost efficiency. She describes routing queries to different models according to complexity. A difficult query might go to GPT-4, while a simpler one could go to Llama at lower cost. Teams can use a routing tool or build their own classifier with logistic regression and a count vectorizer. Caching also reduces the number of LLM calls needed to answer repeated or similar queries.
LLM applications change how software teams test and collaborate
Misra says software engineers can find LLM systems difficult to understand because the execution flow is different and much of the behavior is hidden. It can be hard to track which plugin is called and how much the prompt controls the result. Tests are not deterministic, and changing code or a prompt can fix one problem while degrading the system overall. Token limits add another difference from traditional software engineering. She says close communication and collaboration with software engineers is important.
"What I've seen is generally it's this RAG model, a RAG system, which goes into production or it's RAG is acting as a component of a larger system."02:20
Who should watch
You are building a document-question-answering system and need to understand the production choices around chunking, retrieval, embeddings, and vector storage.
Your RAG prototype works, but you need a way to evaluate retrieval and answer quality before deploying it.
You work with software engineers on an LLM application and need practical guidance on prompt sensitivity, nondeterministic tests, model routing, and cost.
