# Does AgenticRAG Really Work?

Satish Bhambri, Walmart Labs | MLOps Podcast | Episode 351 | 1:01:40
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=RSaiFfM9G3A
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/does-agenticrag-really-work
Published: 2025-12-12
Tags: agents, embeddings, evals, multi-agent, rag

## TL;DR
- Agentic RAG breaks a broad retrieval task into specialized agents that can manage their own context and communicate with other agents.
- Dynamic prompts are built from retrieved schema documents and relationships, which gives language models more useful context for generating SQL and reports.
- Evaluating a RAG agent requires checking document retrieval, dynamic prompt quality, and the final model output as separate stages.

## Summary
Satish Bhambri explains how RAG evolved into agentic RAG systems that use specialized agents for specific data sources and business tasks. A conventional language model can produce syntactically valid SQL while missing the correct tables, joins, or columns. Bhambri describes schema documents, embeddings, and relationship mappings that help an agent retrieve the right context and construct a dynamic prompt before asking an LLM to generate SQL. He connects this design to separation of concerns, data governance, cost choices, and different requirements for internal and consumer-facing systems. The conversation also covers LLM-assisted recommendations at Walmart Labs, where language models extract product features from recipes and other pages before traditional recommender systems use those signals. Bhambri is candid about the difficulty of stale vector data and agent evaluation. His evaluation process checks retrieval, prompt generation, and final answers separately, with recall and LLM-based feedback among the measures he discusses.

## Key ideas
### Agentic RAG specializes retrieval around a particular use case
[05:36](https://www.youtube.com/watch?v=RSaiFfM9G3A&t=336s)
Bhambri says early RAG systems grounded an LLM with broadly relevant information from vector stores. Agentic RAG narrows the work by creating agents for specific use cases, then letting those agents communicate. He compares this with the move from REST services to microservices, where each service is tailored to a narrower responsibility. One agent might work with a marketing database while another handles sales data. This separation makes it easier to add security, enrich data between steps, maintain governance, and choose different levels of quality or cost for internal and external users.

### Dynamic prompts carry the schema context that plain SQL generation lacks
[17:17](https://www.youtube.com/watch?v=RSaiFfM9G3A&t=1037s)
A language model given only a table schema can produce a SQL query, but Bhambri says it may not know which tables to join or which columns connect them. The agent therefore retrieves documents containing schemas and relationships, then uses them to build the prompt passed to the LLM. For a question such as "How did we do this quarter?", the retrieved context can connect sales, products, and customers, identify the relevant date range, and specify which measures to fetch. The SQL is still generated by the LLM, but the prompt gives it a more constrained description of the task.

### Vector database choices depend on freshness, latency, and scale
[18:42](https://www.youtube.com/watch?v=RSaiFfM9G3A&t=1122s)
Bhambri chooses vector infrastructure according to the workload. A daily recommendation job can use a less expensive open-source option, while a pipeline that serves recommendations every 15 minutes needs lower latency and may justify a more expensive system. He mentions Milvus DB and Vertex AI as examples of options with different cost and latency trade-offs. A small reporting use case may need only lightweight file-based similarity search. The same architecture does not fit every agent, because the amount of data, update frequency, and serving requirements differ.

### Routing lets agents stay close to their own data and terminology
[26:18](https://www.youtube.com/watch?v=RSaiFfM9G3A&t=1578s)
For a new system, Bhambri would first examine the data type, the problem, and the governance rules for different locations. He would build small agents for specific use cases and route incoming queries according to their context. An agent focused on sales should not answer a human-resources question using the same interpretation of the data. The routing happens inside the RAG agent during augmentation, before the final generator. This design lets each agent retrieve from its own data set and produce a prompt tuned to that context.

### Schema mappings help agents interpret ambiguous business language
[29:49](https://www.youtube.com/watch?v=RSaiFfM9G3A&t=1789s)
The conversation turns to questions such as "How did we do this quarter?" Bhambri says a useful system has to map vague language to the relevant business concepts. Schema documents can define relationships and glossary mappings, so terms can resolve to sales, revenue, products sold, or employee performance depending on the agent and user context. The date phrase can map to a time range. A sales agent and an HR agent may therefore return different answers to the same words. Bhambri says this context-specific mapping is necessary because an LLM tends to answer even when it does not understand the intended question.

### Index construction determines what semantic retrieval can return
[42:04](https://www.youtube.com/watch?v=RSaiFfM9G3A&t=2524s)
Bhambri explains that retrieval starts before the user asks a question. Documents are embedded and placed in a vector space, with each document capturing data such as a table schema, relationships, or a question-and-answer exchange. A new query is embedded with the same model, then compared with the indexed material using a similarity measure such as cosine similarity, Euclidean distance, or Pearson correlation. The system can tune how many neighboring results it retrieves. Bhambri gives five nearest neighbors as a useful approximation for one small use case, while warning that too much or too little context can both reduce answer quality.

### Keeping indexed information current remains an unresolved cost
[47:16](https://www.youtube.com/watch?v=RSaiFfM9G3A&t=2836s)
When data becomes stale, Bhambri says the direct solution is to re-index the database, although that becomes harder and more expensive as the data grows. A negative example in the documents can tell the system not to use a particular stale policy, but he describes that as a sanity check rather than a scalable replacement strategy. It may work for one or two pieces of data, but it does not solve the problem for a large set of changed documents. He also suggests that parts of a vector database might be re-indexed instead of rebuilding everything.

### LLMs can enrich traditional recommendation systems without replacing them
[51:29](https://www.youtube.com/watch?v=RSaiFfM9G3A&t=3089s)
Bhambri describes a hybrid recommendation approach. An LLM can read a recipe or another product-related page, extract relevant features, and identify products that connect to the user's current interest. Those signals can then enrich a traditional recommender system and help diversify later recommendations. He gives the example of a user viewing a sushi recipe, where related cooking tools might become useful recommendations. The approach can have high latency for online serving, so caching and lightweight models help bring the results into production. The LLM adds context around the item, while the recommender still handles the broader recommendation process.

### RAG evaluation has to separate retrieval, prompting, and generation
[58:09](https://www.youtube.com/watch?v=RSaiFfM9G3A&t=3489s)
Bhambri breaks evaluation into three stages. First, the agent must retrieve the right documents, which can be assessed with recall. Second, the dynamic prompt built from that context must be relevant. Third, the LLM must generate an appropriate result from the prompt. He says early prototypes required manual debugging, while later evaluation uses model settings such as temperature and K, along with LLM feedback on generated content. The feedback module sends one agent's output to another language model for assessment. Bhambri is clear that this process is still developing and does not yet have a single state-of-the-art method.

## Notable quotes
- Satish Bhambri: "The sole of an agent lies in how dynamically generates it prompt which is passed to the large language model." (17:17)
- Satish Bhambri: "We can't stop that." (18:09)
- Satish Bhambri: "It would not know where to join. It would not know how to or which particular columns to join on." (17:30)
- Satish Bhambri: "These three phases are ones like we're still figuring out." (1:01:36)

## Tools & references mentioned
- Waymo
- Attention Is All You Need
- ChatGPT
- SOAP
- RESTful APIs
- microservices
- SQLGlot
- Vertex AI Matching Engine
- Milvus DB
- Facebook AI Similarity Search
- Thompson sampling
- Power BI
- Donae
- Dr. D. Kai
- Raising AI
- Walmart Labs
- Instagram
- YouTube

## Who should watch
- You are building an agent that needs to query databases, generate reports, or interpret internal business terminology.
- Your RAG prototype retrieves plausible documents but still produces incorrect SQL or answers that lack the intended context.
- You run recommendation systems and want to understand where an LLM can add item context without replacing the existing recommender.

## Related talks

- [Supercharging Your RAG System: Techniques and Challenges](https://mlopstalks.com/talks/supercharging-your-rag-system-techniques-and-challenges) (Tengyu Ma, Voyage AI and Stanford University, 40:20)
- [RagSys: RAG is just RecSys in Disguise](https://mlopstalks.com/talks/ragsys-rag-is-just-recsys-in-disguise) (Chang She, LanceDB, 12:45)
- [RAG Has Been Oversimplified](https://mlopstalks.com/talks/rag-has-been-oversimplified) (Yujian Tang, Zilliz, 48:56)
- [Retrieval Augmented Generation](https://mlopstalks.com/talks/retrieval-augmented-generation) (Syed Asad, KiwiTech, 44:10)
- [A Survey of Production RAG Pain Points and Solutions](https://mlopstalks.com/talks/a-survey-of-production-rag-pain-points-and-solutions) (Jerry Liu, LlamaIndex, 30:00)