# Using Vector Databases: Practical Advice for Production

Sam Partee, Redis | LLMs in Production 2023 | 29:58

Source: https://www.youtube.com/watch?v=tchI22onVYE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/using-vector-databases-practical-advice-for-production
Published: 2023-07-03
Tags: caching, embeddings, feature-engineering, feature-stores, rag

## TL;DR
- Vector databases let applications retrieve changing context without retraining and redeploying a language model.
- Production systems need deliberate choices about chunking, prompts, embeddings, indexes, metadata, access control, and cost.
- Semantic caching, feature injection, guardrails, and selective long-term memory can reduce latency, model calls, and irrelevant context.

## Summary
Sam Partee explains how vector embeddings turn unstructured data into searchable numerical representations, then focuses on the production patterns he has deployed with customers and in practice. Context retrieval can provide current information to a language model, support access controls, and avoid the speed and cost of fine-tuning. He covers hypothetical document embeddings, feature injection from an online feature store, semantic caching, guardrails, and selective long-term memory. The practical advice is to inspect what integrations do with documents and prompts, test chunk sizes and retrieval settings, plan index architecture before scaling, and keep metadata close to vectors when possible. He also discusses filtering data by tenant, choosing between separate and grouped indexes, projecting costs, testing performance at mock scale, and avoiding extra network hops. The final examples show AWS, Azure, and on-premise architectures for retrieval applications.

## Key ideas
### Embeddings turn meaning into searchable numerical representations
[02:12](https://www.youtube.com/watch?v=tchI22onVYE&t=132s)
Sam describes a vector embedding as a list of numbers where each dimension says something about unstructured data such as audio, text, or images. These vectors are usually dense and high-dimensional. Similarity search compares vectors with measures such as cosine similarity, so a sentence about a happy person can match another sentence with the same meaning even when the words differ. A vector database stores the embeddings and supports searches against them. Sam also describes Redis vector search, including HNSW, FLAT, L2, cosine, inner product, hybrid queries, and JSON support.

### Context retrieval keeps changing information outside the model
[06:42](https://www.youtube.com/watch?v=tchI22onVYE&t=402s)
The most common pattern is retrieving relevant information from a knowledge base and adding it to a prompt for a chatbot, question-answering system, or recommendation system. Sam says this is faster and cheaper than fine-tuning and allows a changing source of information to be reflected without retraining and redeploying the model. Retrieval can also support access restrictions. For example, different analyst groups can receive different documents through role-based access control, instead of relying on a fine-tuned model not to reveal data it has seen.

### A second retrieval pass can improve weak context
[09:09](https://www.youtube.com/watch?v=tchI22onVYE&t=549s)
Sam explains a hypothetical document embeddings approach that first retrieves context and then summarizes or transforms that context before passing it to the answering model. This requires two generative-model calls and can take seconds. His suggested pattern is to run a normal search and the slower fallback asynchronously. If the first search returns useful context above a chosen similarity threshold, the system can use it. If it returns nothing useful, the second approach can provide another route to relevant context.

### Feature stores can add live user and product information to prompts
[10:15](https://www.youtube.com/watch?v=tchI22onVYE&t=615s)
When a vector database and an online feature store are available in the same flow, an application can retrieve semantic context and current entity features together. Sam gives an e-commerce chatbot example. If a user asks about a previous order, the system can identify the request, look up information about the user or product, and add details such as recent purchases or rewards status to the prompt. The language model did not know those facts beforehand. With Redis, Sam says vectors, metadata, and features can be co-located in the same infrastructure.

### Semantic caching avoids repeated model calls for similar questions
[13:28](https://www.youtube.com/watch?v=tchI22onVYE&t=808s)
A normal cache returns a result for the same exact key, but semantically similar questions can have different hashes. Sam describes embedding a query and comparing it with previous queries using a similarity threshold. A question such as asking about a product can then reuse an answer for a slightly different wording. This removes the language-model call, reduces monetary and computational cost, and improves queries per second. For fixed FAQ content, teams can prepare the questions and answers in advance and cache the responses.

### Vector lookups can constrain model behavior
[15:39](https://www.youtube.com/watch?v=tchI22onVYE&t=939s)
Sam discusses guardrails as a further use of retrieval. A system can define a default answer when no context is found, choose another path, or follow a precomputed directed acyclic graph of options. Fast vector lookups can support several branches before the application calls the language model. He compares this with bowling bumpers: the model can still move quickly, but predefined paths reduce the chance that it goes into an unwanted direction. He mentions NVIDIA NeMo Guardrails and its Colang definitions as a related example.

### Longer context windows do not remove the need for selective memory
[17:48](https://www.youtube.com/watch?v=tchI22onVYE&t=1068s)
Even when a model accepts a large context window, Sam advises against putting the entire previous conversation into every prompt. Long histories can contain irrelevant information and increase computational cost. A search based on a user, topic, or current query can retrieve more useful memories. Teams need to decide how many tokens belong in each embedding and how many context pieces to retrieve for each prompt. These choices affect the quality and cost of the resulting application.

### Production quality depends on testing defaults and planning index architecture
[19:54](https://www.youtube.com/watch?v=tchI22onVYE&t=1194s)
Sam criticizes applications that rely on library defaults without inspecting the resulting chunks or prompts. Teams should check how documents are split, what enters the prompt, and how many tokens are used. For bounded tasks such as FAQs, they can test context-window size, retrieved pieces, and tokens per embedding with grid searches across data folds. Index architecture also needs advance planning. A multi-tenant system might filter one large index by store metadata or use many smaller indexes. The choice affects recall, query performance, memory, and billing.

### Mock scale testing exposes metadata, network, and cost problems early
[23:45](https://www.youtube.com/watch?v=tchI22onVYE&t=1425s)
Sam recommends projecting costs, testing performance, and creating fake schemas and embeddings before production growth. Separate metadata can require an extra network hop, while storing metadata with the vectors can avoid that lookup. He says Redis does not charge per index, so multiple smaller indexes can work for some workloads, while larger or grouped indexes may be better beyond a workload-dependent scale. He closes with AWS, Azure, and on-premise example architectures for retrieval applications.

## Notable quotes
- "Vector embeddings are essentially lists of numbers." (02:28)
- "The whole point here is that you have some question and answer loop or some chat bot or some recommendation system, and your goal is to retrieve contextually relevant information from within your knowledge base." (07:02)
- "You'd be surprised how you can use worse models for the generative side and actually improve upon these factors." (21:01)
- "So think through all of that ahead of time, project your cost, test performance, and mock scale." (24:30)

## Tools & references mentioned
- Redis
- Hugging Face
- OpenAI
- Cohere
- Sequoia
- Redis Adventures
- Featureform
- NVIDIA NeMo Guardrails
- LangChain
- GTC
- Eric Bernhardin
- Amazon SageMaker
- AWS
- Azure
- Terraform
- Party.io

## Who should watch
- You are building retrieval-augmented applications and need practical decisions about chunking, prompts, embeddings, and cache thresholds.
- Your system has multiple tenants, sensitive documents, an online feature store, or a growing number of indexes.
- You want to estimate vector-search cost and performance before committing to a production architecture.

## Related talks

- [Why Purpose-built Vector Databases Matter for Your Use Case](https://mlopstalks.com/talks/why-purpose-built-vector-databases-matter-for-your-use-case) (Frank Liu, Jiang Chen & Yujian Tang, Zilliz, 59:01)
- [Information Retrieval & Relevance](https://mlopstalks.com/talks/information-retrieval-relevance) (Daniel Svonava, Superlinked, 56:05)
- [Vector Databases and Large Language Models](https://mlopstalks.com/talks/vector-databases-and-large-language-models) (Samuel Partee, Redis, 13:10)
- [Vector Similarity Search at Scale](https://mlopstalks.com/talks/vector-similarity-search-at-scale) (Dave Bergstein, Pinecone, 49:48)
- [Democratizing AI](https://mlopstalks.com/talks/democratizing-ai) (Yujian Tang, Zilliz, 54:18)