Vector embeddings turn text, audio, and images into numerical representations that can be compared by semantic similarity.
2
A vector database can retrieve current, confidential, or proprietary context for a large language model without fine-tuning the model.
3
Semantic caching can reuse an answer for a sufficiently similar query, reducing computation and speeding up applications.
Summary
Samuel Partee explains how vector embeddings and vector databases extend large language models beyond the information in their training data. Embeddings represent unstructured data as lists of numbers, and similarity search finds stored content that is close to a query using measures such as cosine similarity. A vector database can then provide relevant context to a language model at generation time. Partee describes three uses: retrieving external context for question-answering systems, storing selected chat history as long-term memory, and caching answers for semantically similar queries. These methods can handle confidential documents and rapidly changing information without repeatedly fine-tuning a model. He also presents Redis as a vector database with secondary indexing for hash and JSON documents, plus flat and HNSW index types. The talk is brief and moves quickly, but its examples give a practical picture of how retrieval fits around a language model.
Embeddings represent unstructured data as comparable numerical vectors
Partee describes vectors as lists of numbers produced by machine learning models from audio, text, or images. Each number captures some part of the input, and the complete list forms a high-dimensional embedding. Models from OpenAI and Hugging Face make it easy to extract these representations through APIs. The useful property is that semantically related inputs can be close together even when they do not share every word. This lets a search system compare the meaning of a query with stored content rather than relying only on exact keyword matches.
Cosine similarity finds the stored sentence closest to a query
In Partee's example, three semantic vectors form a search space and the query is "a happy person." The system compares the query vector with the stored vectors by calculating distance. It uses cosine similarity to measure how close the vectors are. The sentence "that is very happy person" is judged more similar than alternatives that happen to contain words such as "happy". The example shows why embeddings can capture semantic relationships instead of treating each word as an isolated match.
A vector database operationalizes similarity search for applications
Partee says a vector database stores embeddings and provides a query interface for applications. In his Redis example, embeddings can be stored alongside a secondary index, allowing production systems to perform operations such as create, read, update, and delete. Redis with Redis Search supports secondary indexing on hash or JSON documents. Partee names flat and hierarchical navigable small world index types, and says Redis was also working on a GPU index with NVIDIA.
Context retrieval gives a language model current or private information
Large language models may have broad training data, but they do not know a company's confidential documents, proprietary information, or rapidly changing facts. In a question-answering system, the question becomes an embedding, which searches the vector database for semantically similar context. That retrieved context is then supplied to the language model during generation. Partee says this approach costs less than fine-tuning, supports real-time updates, and avoids putting sensitive information into the fine-tuning process.
External retrieval avoids repeatedly fine-tuning for fast-changing data
Partee uses stock-market information as an example of data that changes too quickly for repeated model fine-tuning. A system that recommends trades from current news would need an external knowledge base that updates at the pace of the market. Retrieval lets the application use the latest stored context when generating an answer. The model can therefore work with information that was absent from its original training set without rebuilding its weights each time the information changes.
Vector storage can provide selective long-term memory for chatbots
For chatbot memory, Partee discusses a project called ChatGPT Memory. The method addresses topic changes across multiple user sessions and the limited context length of many models. Instead of sending the entire conversation, the system can isolate the last K messages relevant to a particular message or session. Partee presents this as a way to provide a chatbot with targeted historical context while keeping unrelated conversation out of the prompt.
Semantic caching reuses answers for sufficiently similar queries
Partee describes GPTCache as a system that stores answers in a semantic vector database. When a new query is similar enough to a previous query, and it passes a chosen threshold, the application can return the cached answer instead of calling the language model again. This can reduce computational and monetary costs and speed up applications because language models are slow. He also mentions work with NVIDIA on a related Trident Response Cache.
