LLM applications struggle with missing domain knowledge, stale training data, hallucinations, and the cost of repeating queries.
2
The CBP framework combines an LLM, a vector database, and application code to add domain knowledge to an LLM application.
3
Vector databases store embeddings and support production features such as metadata filtering, hybrid search, backups, high availability, sharding, and distributed querying.
Summary
Yujian Tang explains why LLM applications need more than a language model. Models can lack domain knowledge, contain outdated information, hallucinate plausible code, and become expensive when they answer repeated questions. His CBP framework treats the LLM as a processor, a vector database as storage, and application code as the interface between them. He uses OSS Chat as an example. The application retrieves documentation, creates embeddings, and uses that information to answer questions about open-source software. A cache can avoid sending repeated questions through the LLM. Tang then describes vector embeddings, approximate nearest-neighbor indexes, and the production features that distinguish a vector database from a search library. He closes with Milvus architecture, including separate query, data, and index nodes, parallel work on 512 MB data chunks, and a short demonstration of documentation-grounded answers.
LLM applications have data and cost problems before they have scaling problems
Yujian says that LLM applications often lack the domain knowledge a user needs and do not contain up-to-date data. He gives GPT-3 as an example, saying that its training data ended in September 2021. Repeated queries also create a cost problem, especially when an application runs them frequently. These limitations matter when a team moves from trying an LLM in a chat window to building an application around it. The application needs a way to supply current, specialized information without retraining the model for every change. It also needs to avoid paying for the same generation repeatedly.
LLMs generate likely continuations, so plausible answers can still be wrong
Yujian describes language models as stochastic neural networks that predict the most likely next token from the tokens that came before it. He uses a made-up Milvus answer to show the problem. ChatGPT can produce Python-looking code for connecting to Milvus, but the code may not match how the software actually works. The answer looks credible because its syntax is familiar. It is still a hallucination. Yujian frames the practical response as adding domain knowledge to the LLM application. The model remains the generator, while retrieved information gives it facts and examples that are specific to the software or subject.
The CBP framework divides an LLM application into compute, storage, and interface
Yujian presents the CBP framework as ChatGPT, vector databases, and prompt or application code. ChatGPT, or another LLM, acts like the processor. The vector database acts like storage for domain information. The application code connects the two and provides the interface. He compares this arrangement with a general-purpose computer, where a processor works with memory and code. The framework is meant for production LLM applications rather than an isolated model call. It gives each part a distinct job: the LLM generates language, the database supplies relevant information, and the surrounding code decides how users, retrieval, and responses interact.
OSS Chat grounds answers in documentation and caches repeated questions
OSS Chat demonstrates the CBP approach with questions about open-source software such as Milvus and PyTorch. The system collects documents, creates embeddings, and uses the source material to answer questions. Yujian says the application can also generate question and answer examples from the documents. When a user asks how to query Milvus, the system can return an answer based on the actual documentation and code rather than a generic model completion. He also describes GPTCache, which was built as part of OSS Chat. If many users ask the same question, the system can retrieve a stored answer instead of sending every request through the LLM again.
Embeddings let an application search domain knowledge by meaning
Yujian explains that vector databases provide semantic search over domain knowledge through vector embeddings. An embedding is a numeric representation of an object such as a word, image, or document. He uses the familiar example where the relationship between woman and queen is similar to the relationship between man and king, while noting that real systems use vectors with hundreds of dimensions. In practice, a deep learning model processes the source object and the application takes the values from the second-to-last layer as its embedding. Those embeddings go into a vector database, where a new query can be compared with stored information by distance or similarity.
A production vector database adds operational features around vector search
Yujian defines a vector database as a system built to store, index, and query large quantities of vector embeddings. He distinguishes it from libraries such as FAISS and from individual index methods such as HNSW. A production database can filter results by metadata, combine dense and sparse or text search, provide backups and high availability, and support sharding, parallel search, lifecycle management, multi-tenancy, and GPU acceleration. He presents Milvus as focused on large collections, including billion-scale storage. The point is that approximate-nearest-neighbor search alone is not the whole production problem. Teams also need storage, scaling, recovery, and ways to control which records are searched.
Milvus distributes query, data, and indexing work across storage and compute components
Yujian describes Milvus as using a distributed-system back end with a load balancer, query nodes, data nodes, and index nodes. Query nodes handle database queries, data nodes keep working data in memory, and index nodes build indexes. The components can run together, but separating their jobs can improve performance. Milvus stores data in 512 MB chunks instead of one large block. Queries can then work on those chunks in parallel rather than scanning a single 1, 10, 30, or 100 GB block. The data is kept in persistent storage such as S3, MinIO, or Azure Blob Storage. He says this architecture usually removes the need to re-index after every change.
"The solution to these hallucinations is basically to inject domain knowledge into large language models, really to inject domain knowledge on top of them, on top of large language models, but into LLM applications."09:19
Who should watch
You are building an LLM application that needs current company or product documentation instead of relying only on model training data.
Repeated LLM calls are increasing costs, or users are receiving plausible answers that contain incorrect code.
Your team is choosing between a vector search library and a production vector database, and needs to understand the operational differences.
