# The Vector Search Engine Powering AI Agents

Thierry Damiba, Qdrant | MLOps Community | 27:45
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=VDFiHwk_C0Y
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-vector-search-engine-powering-ai-agents
Published: 2025-11-26
Tags: agents, caching, embeddings, search

## TL;DR
- AI agents need memory, access to fragmented multimodal data, and infrastructure that can handle production traffic.
- Qdrant combines dense, sparse, and hybrid search with metadata filters, real-time upserts, semantic caching, and multimodal vectors.
- An agent uses an LLM in a loop to choose its next action, while Qdrant supplies search, context, and fast updates inside that workflow.

## Summary
Thierry Damiba explains why vector search is useful inside AI agents. Agents need to remember past interactions, search across separate enterprise data sources, and handle many writes, users, and access rules in production. Qdrant supports this with dense, sparse, and hybrid search, metadata filtering, real-time upserts, multimodal vectors, cloud inference, and semantic caching. Damiba distinguishes an agent from a fixed chain: an agent uses an LLM in a loop to choose tools or actions. He shows how an agent can expand a query, extract filters, search internal knowledge, call a web search, and use an LLM or VLM to judge results. A live demo combines a Qdrant knowledge base, semantic cache, and Perplexity web search. The talk also explains Qdrant's storage and compute design, its focus on write-heavy workloads, and its integrations with agent frameworks.

## Key ideas
### AI agents have existed for decades, but modern systems make them useful at a larger scale
[00:43](https://www.youtube.com/watch?v=VDFiHwk_C0Y&t=43s)
Damiba places the first AI agents in the 1960s, naming ELIZA, Shakey the robot, and Dendral. ELIZA was an early chatbot, while Shakey could learn from its environment. He argues that agents are not a new idea. The recent change comes from better language models, search systems, and workflows that can plan several actions. TripAdvisor's Trip Builder illustrates this shift. Users provide a destination, travel companions, dates, and interests, and the system creates an itinerary instead of returning separate search results.

### Vector search gives agents memory and access to fragmented data
[06:46](https://www.youtube.com/watch?v=VDFiHwk_C0Y&t=406s)
Damiba identifies memory loss, fragmented knowledge, and production scale as problems vector search can address. An agent can retain useful interactions with the current user and with other users. In an enterprise, the relevant information may be spread across hotel images, text reviews, and videos. Embeddings place different modalities into a shared semantic space, so text can search images, images can search text, and videos can be searched by their text, image, or audio content.

### Qdrant combines search controls with real-time agent features
[05:48](https://www.youtube.com/watch?v=VDFiHwk_C0Y&t=348s)
Qdrant supports metadata filters, sparse queries, dense queries, and hybrid queries so an application can narrow results by meaning and exact attributes. Damiba also describes multi-tenancy, real-time upserts, CPU optimization, and semantic caching. A semantic cache can return an answer for a sufficiently similar earlier query instead of calling the LLM again. He says Qdrant can use existing embeddings, its FastEmbed provider, or cloud inference.

### An agent chooses actions in a loop, while a chain follows a fixed pattern
[13:08](https://www.youtube.com/watch?v=VDFiHwk_C0Y&t=788s)
Damiba separates simple chains from agents. Stuffing, map-reduce, refine, and map-rank apply predefined steps to retrieved data. He defines an agent as a system that uses an LLM in a loop to choose its next action, often from tool descriptions. A routing agent might choose Qdrant, web search, or another tool. A more advanced agent can search first, then decide whether the result is sufficient or whether it needs an external API or web search.

### Agents can improve search before and after retrieval
[17:07](https://www.youtube.com/watch?v=VDFiHwk_C0Y&t=1027s)
Damiba gives several examples of how an LLM can help a vector search workflow. It can expand a short or misspelled query such as 'men's denim size large'. It can extract structured filters from requests such as a blue shirt, a price below 100 dollars, or size large. An LLM or VLM can also score candidate items and help select the result shown to the user. These steps let an agent turn a vague request into a more precise search.

### Qdrant is designed for write-heavy, low-latency vector search
[18:34](https://www.youtube.com/watch?v=VDFiHwk_C0Y&t=1114s)
Damiba describes Qdrant as a search engine rather than a general database. He says its local coupling of storage and compute avoids some cold-start latency. He also focuses on workloads with frequent writes, such as tracking possible financial fraud. Transactions may need to be inserted, updated, or marked as confirmed fraud as new information arrives. His argument is that agent systems can fall apart in production if their vector infrastructure cannot handle this write activity.

### A cache-first workflow can combine private knowledge with web search
[23:03](https://www.youtube.com/watch?v=VDFiHwk_C0Y&t=1383s)
In the live demo, the agent first checks a semantic cache for a similar previous query. If there is no match, it searches an internal Qdrant collection containing Wikipedia articles. If the answer is not there, it uses Perplexity for web search. Damiba shows a query where the web search took almost seven seconds while the Qdrant portion took five milliseconds. A repeated query returns from the cache, avoiding another search and LLM call. The same workflow can answer from internal data or retrieve current information from the web.

## Notable quotes
- Gary Tan: "Agents without memory of me and what I care about and all the context around me are just not as useful." (06:46)
- Thierry Damiba: "What I consider to be an agent is something that uses LLMs in a loop to choose the next action to perform." (13:51)
- Thierry Damiba: "The beauty of a vector search engine is taking that unstructured data, turning it into numbers." (08:38)
- Thierry Damiba: "Quadrant is a search engine." (18:34)

## Tools & references mentioned
- Qdrant
- TripAdvisor
- Trip Builder
- ELIZA
- Shakey the robot
- Dendral
- Gary Tan
- Y Combinator
- BERT
- FastEmbed
- Perplexity
- Wikipedia
- CrewAI
- OpenAI Swarm
- AutoGen
- Gemini
- Mistral
- AWS

## Who should watch
- You are building an agent that needs to search private documents, call web APIs, or combine several tools in one workflow.
- Your prototype works with a small dataset, but you expect frequent updates, multiple users, or write-heavy production traffic.
- You need multimodal search, semantic caching, or a clearer distinction between a fixed retrieval chain and an agent.

## Related talks

- [Agents as Search Engineers](https://mlopstalks.com/talks/agents-as-search-engineers) (Santoshkalyan Rayadhurgam, Meta, 29:38)
- [Using Vector Databases: Practical Advice for Production](https://mlopstalks.com/talks/using-vector-databases-practical-advice-for-production) (Sam Partee, Redis, 29:58)
- [The Next Evolution of AI Agents](https://mlopstalks.com/talks/the-next-evolution-of-ai-agents) (Alon Horev, Vast Data, 13:38)
- [Vector Similarity Search at Scale](https://mlopstalks.com/talks/vector-similarity-search-at-scale) (Dave Bergstein, Pinecone, 49:48)
- [Real-time features, AI search, Agentic similarities](https://mlopstalks.com/talks/real-time-features-ai-search-agentic-similarities) (Varant Zanoyan & Nikhil Simha Raprolu, Zipline AI, 29:28)