# RAG Has Been Oversimplified

Yujian Tang, Zilliz | MLOps Podcast | Episode 206 | 48:56
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=tz8xi3u0lu8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/rag-has-been-oversimplified
Published: 2024-01-23
Tags: embeddings, evals, observability, rag

## TL;DR
- RAG does not remove hallucinations because the language model still predicts answers from statistical patterns, even when it receives retrieved context.
- A production RAG application needs evaluation, observability, source citations, data pipelines, and prompts that engineers can adjust.
- RAG is a poor fit for structured data tasks and some similarity-search applications, where a database query or vector search is enough.

## Summary
Yujian Tang explains why the standard RAG diagram hides important engineering choices. An application usually combines an embedding model, a vector database, and a language model, but those components do not have to come from the same provider or use the same model. The language model interprets a user's question, creates a database query, receives relevant context, and turns that context into an answer. RAG can reduce hallucinations about proprietary data, but it cannot guarantee correct answers. Production systems need evaluation, observability, citations, data-ingestion pipelines, and prompt controls. Tang also discusses chunk size, overlap, and storage choices, which depend on the data and the way users interact with the application. He gives structured user data and a fashion-search project as examples where RAG or an LLM may be unnecessary. Multimodal RAG adds images, audio, and video, along with the risk of compounded hallucinations and expensive processing.

## Key ideas
### Embedding models and language models have different jobs
[05:00](https://www.youtube.com/watch?v=tz8xi3u0lu8&t=300s)
Yujian Tang separates the embedding model from the language model in a typical RAG application. The embedding model turns internal or proprietary data into numerical vector representations. The language model interprets questions and generates readable answers. They do not have to be the same model. Tang says developers can start with a basic model from Hugging Face, but specific enterprise data may require training the embedding model on the right type of data. He also explains that embeddings use the output from the second-to-last layer, because the goal is to capture the input's semantics rather than the model's final prediction.

### A RAG request turns a natural-language question into a database query
[10:03](https://www.youtube.com/watch?v=tz8xi3u0lu8&t=603s)
Tang describes a suggested request flow behind a question-answering application. The language model first interprets the user's question and its surrounding instructions. For a question about black holes in the book Cosmos, it can identify the book and topic as filters. The application then makes an API, HTTP, or gRPC call to the vector database. Retrieved records can include the book, section, chapter, and author. The language model receives those results with the original question and turns them into an answer for the user. The design of this flow depends partly on how users are expected to interact with the application.

### RAG reduces some hallucinations, but it does not make answers correct
[17:29](https://www.youtube.com/watch?v=tz8xi3u0lu8&t=1049s)
Tang rejects the idea that retrieved context eliminates hallucinations. Injecting a company's data can reduce the chance that the model invents facts about that data, but the model still predicts an answer statistically. A strange training example or a contradiction between the retrieved context and training data can produce a wrong result. For production use, Tang recommends evaluation and observability. Teams should check whether retrieved context is relevant, since semantic similarity alone is insufficient, and they should cite sources so users can inspect where an answer came from.

### Citations trade extra storage for more grounded answers
[19:50](https://www.youtube.com/watch?v=tz8xi3u0lu8&t=1190s)
Tang explains that citations require storing the source text alongside the embeddings. This uses more data, but it gives the application a way to return the underlying context and keep its answer closer to the retrieved source. He describes writing a pull request for LlamaIndex to make citation support available. He also distinguishes context, similarity, and relevance. The sentence "a banana a day keeps the chiropractor away" may resemble a question about what keeps the doctor away, yet it is not relevant to that question. Similarity is therefore not enough to judge retrieval quality.

### A production RAG stack includes ingestion and prompt controls
[24:16](https://www.youtube.com/watch?v=tz8xi3u0lu8&t=1456s)
The RAG stack grows beyond an embedding model, language model, and vector database. Tang adds evaluation and guardrails for checking output before it reaches users. He also describes data pipelines that move new source data through embedding and ingestion steps into the vector database. Zilliz has been building pipelines for this purpose, including an open-source embedding model in its pipeline. Tang says production teams may also need prompt tooling and direct access to prompts, because developers will want to adjust them rather than accept framework defaults.

### Chunking should follow both the source data and the user experience
[30:07](https://www.youtube.com/watch?v=tz8xi3u0lu8&t=1807s)
Tang treats chunking as a pre-processing decision that comes before embedding. A conversational chatbot may need small chunks so it can retrieve specific pieces of information. An application that helps write longer passages may benefit from larger blocks of text. The source format matters too. Conversations can be split around message boundaries, while large documents may need larger chunks. Some documents also need overlap, such as the final sentence of one paragraph and the first sentence of the next, so retrieval does not lose surrounding meaning.

### Some applications need search or a database instead of RAG
[35:23](https://www.youtube.com/watch?v=tz8xi3u0lu8&t=2123s)
Tang gives structured user data as a clear case where RAG is the wrong tool. If a team wants to map users by predefined characteristics, the data is structured, so a database and an automation are more appropriate. He also describes a fashion project that uses image segmentation and vector similarity search to compare clothing with celebrity outfits. The application does not need an LLM to describe every image or generate an answer. Tang uses this example to show that a vector database can be useful without turning the whole application into a RAG system.

### Multimodal RAG can compound errors across images, audio, and text
[43:08](https://www.youtube.com/watch?v=tz8xi3u0lu8&t=2588s)
Tang expects multimodal RAG to combine images, audio, video, and scripts, but he is careful about its limits. An LLM may hallucinate when interpreting an image, then hallucinate again when using that interpretation in a retrieval or answer step. Video search is also expensive because a video contains many frames, audio, and other data that must be aligned. He mentions possible uses such as searching a movie or finding the episode behind a clip in a music video. He expects the area to become usable, while acknowledging that he does not yet have a solution for compounded hallucinations.

## Notable quotes
- Yujian Tang: "Embeddings models and LLMs are not the same thing." (05:01)
- Yujian Tang: "Injecting your data does it reduces a certain type of hallucination, but that doesn't mean that it's not possible for the RAG to still put out the wrong answer." (17:48)
- Yujian Tang: "Just because something is semantically similar doesn't mean that it may necessarily be semantically relevant." (19:28)
- Yujian Tang: "You don't need an LLM to convert your picture of a dress into the description that this is a picture of a dress if you have pictures of dresses in your vector database." (38:55)
- Yujian Tang: "I think one of the main issues is going to be this issue of reducing compounded hallucinations." (43:40)

## Tools & references mentioned
- Zilliz
- Zilliz Cloud
- Zilliz pipelines
- LlamaIndex
- LangChain
- Hugging Face
- Milvus
- GPT
- Mixtral
- Perplexity
- Falcon
- Llama
- Taylor Swift
- Arize

## Who should watch
- You are building a RAG prototype and need to understand what sits behind the simple embedding-model, vector-database, and LLM diagram.
- Your RAG answers look plausible but you have not added retrieval evaluation, citations, observability, or controls for updating source data.
- You are deciding whether a use case needs RAG at all, especially when the data is structured or the task is straightforward similarity search.

## Related talks

- [A Survey of Production RAG Pain Points and Solutions](https://mlopstalks.com/talks/a-survey-of-production-rag-pain-points-and-solutions) (Jerry Liu, LlamaIndex, 30:00)
- [Supercharging Your RAG System: Techniques and Challenges](https://mlopstalks.com/talks/supercharging-your-rag-system-techniques-and-challenges) (Tengyu Ma, Voyage AI and Stanford University, 40:20)
- [RagSys: RAG is just RecSys in Disguise](https://mlopstalks.com/talks/ragsys-rag-is-just-recsys-in-disguise) (Chang She, LanceDB, 12:45)
- [Retrieval Augmented Generation](https://mlopstalks.com/talks/retrieval-augmented-generation) (Syed Asad, KiwiTech, 44:10)
- [The Real E2E RAG Stack](https://mlopstalks.com/talks/the-real-e2e-rag-stack) (Sam Bean, Rewind.ai, 1:10:07)