Vector databases give LLM applications access to current and private data without retraining the model.
2
A production LLM application should query the vector database first and call the language model only when the retrieved data is not close enough.
3
Vector databases are a poor fit for data that can be handled with key-value pairs or that has no need for semantic similarity.
Summary
Yujian Tang explains why vector databases became widely discussed after ChatGPT. LLMs have fixed training cutoffs, while companies need to use current, private, and domain-specific data. A vector database can store embeddings for documents, questions, and answers, then retrieve relevant material for an LLM application. Yujian describes an approach used in OSS Chat, where an LLM generates possible questions from documentation and the application stores those questions with their answers. He also explains how distance thresholds can stop irrelevant retrievals and prevent unnecessary LLM calls. The conversation covers dirty data, stale data, metadata filters, caching repeated questions, smaller language models, and the limits of automatic systems. Yujian also describes Milvus, Milvus Lite, and Zilliz Cloud, focusing on their different levels of operational complexity and access. His advice is practical: understand the data, choose a distance threshold that fits it, and do not use a vector database when ordinary structured storage is enough.
Vector databases became mainstream because LLM applications need current and private data
Yujian says vector databases were already used for semantic search, image search, and product recommendations before ChatGPT. LLM applications brought them into wider discussion because models have training cutoffs and cannot know a company's recent or private information. An application can inject current data and domain documents through a vector database instead of retraining the model. Yujian also describes caching repeated questions, such as call-center FAQs, so the system can return an existing answer without calling the LLM again. He sees these as common reasons a production LLM application needs a vector store.
Large prompts can work, but their token cost changes the design
Demetrios Brinkmann asks about the trade-off between retrieving information from a vector database and putting a very large amount of text into a prompt. Yujian agrees that large-context models can accept more material, but questions whether an application should spend 100,000 tokens on every interaction with Claude. He says tools such as LlamaIndex and LangChain help keep prompts within reasonable token limits. In his view, the vector database reduces the amount of information sent to the model, while caching can avoid repeated calls for questions that have already been answered.
Smaller language models can handle many enterprise retrieval tasks
Yujian explains that language models are statistical neural networks, so major changes in the world can require many examples if developers try to update the model through fine-tuning. A vector store can hold changing information between the user and the LLM. The model can then interpret the question, reformulate a query, and present retrieved material conversationally. He expects applications to switch among models more easily because many enterprise tasks involve asking about company data and producing a clear response. He says several open-source models can already handle this type of work when used correctly, although judging model quality remains mostly qualitative.
A retrieval application can store generated questions alongside answers
Yujian describes the architecture of OSS Chat, a demo for asking questions about open-source documentation. The process starts by collecting and chunking documents. An LLM generates possible questions about each document, and the application stores the question and answer together in the vector database. When a user asks about building a convolutional neural network with PyTorch, the system searches for the generated question that is closest to the user's wording and returns the associated tutorial or code. Yujian says storing only answer text may produce weaker retrieval, because the user's question can be closer to a generated question than to the answer itself.
Metadata filters handle some questions that semantic search alone cannot answer
Demetrios raises a question about whether John worked on a project before Henry joined the team. Yujian says Milvus supports metadata searches, so an application can filter records by project and date before or alongside vector retrieval. The quality of the result depends on how the application parses the user's request and what information exists in the documents. A language model may be able to infer an answer from team membership records, but the application owner has to decide how to handle questions the model cannot parse or that are not directly represented in the data.
A distance threshold can prevent unnecessary LLM calls
Yujian recommends querying the vector database first and checking whether the nearest result is within an acceptable distance. He explains that embeddings are numerical representations of data and that similarity can be measured with metrics such as cosine similarity or L2 distance. A vector database will return a result whenever it has a vector, even if that result is unrelated, so the application must define when a result is too far away. If the result fails that threshold, the application can call the LLM or return that it does not have the information. The right threshold depends on the application's data.
Structured data often belongs in a key-value store
Yujian says a vector database is unnecessary when the important distinctions can be expressed as key-value pairs. His example is a catalog of cans with fields such as flavor and water type. Filtering for lime, grapefruit, sparkling water, or still water does not require semantic similarity. He also warns that embedding short field-value strings may work poorly because many embedding models are trained on sentences and paragraphs. Converting structured records into embeddings can add complexity without improving the query. The storage and retrieval method should match the form of the data and the question being asked.
Dirty and stale data cause many generative AI failures
Yujian says the most common problems he sees in generative AI applications involve data. People may place CSV data into a vector database without cleaning it, even though the records contain special characters, separators, and independent clauses rather than complete sentences. That format can produce poor embeddings and weak retrieval. He also asks how applications will keep their vector data current, whether through batch pipelines, serving pipelines, or event-based processing. His broader advice is to know the data closely. A model cannot answer well about information that was absent from its training or retrieval data, or that was supplied in a form the system cannot interpret.
Milvus offers different ways to make vector search accessible
Yujian describes Milvus as an open-source vector database designed to work at large scale, with users such as Walmart and eBay. He explains that its query, data, and index nodes handle different parts of the system, while segments are sealed and stored after reaching 512 megabytes. Searches can run across segments in parallel instead of re-indexing all data after every addition. Milvus Lite is easier to start from a notebook, and Zilliz Cloud provides a managed option without requiring users to host or scale the service themselves. Yujian connects this access to education, simpler tooling, and developer advocacy.
"If you have something where your primary differentiator is something that can be stored in key-value pairs, then you don't need a vector database."Yujian Tang33:41
Who should watch
You are building an LLM application that needs current or private documents and want to understand where vector retrieval fits.
Your application is returning irrelevant answers, and you need practical guidance on chunking, embeddings, filters, thresholds, and data cleaning.
You are deciding between Milvus, Milvus Lite, a managed vector database, or a simpler key-value store.
