# Embeddings and Retrieval for LLMs: Techniques and Challenges

Anton Troynikov, Chroma | LLMs in Production 2023 | 35:19

Source: https://www.youtube.com/watch?v=kZeOPapQ8yM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/embeddings-and-retrieval-for-llms-techniques-and-challenges
Published: 2023-07-05
Tags: embeddings, evals, rag, reranking

## TL;DR
- Retrieval gives an LLM programmable memory by finding documents near a query in embedding space and adding them to the model's context.
- Embedding model choice, document chunking, and retrieval relevance still require application-specific testing because there are no settled answers.
- Good retrieval systems combine semantic search with document structure, metadata filters, keyword search, re-ranking, and feedback from users.

## Summary
Anton Troynikov explains how embeddings let an application add external context to an LLM. Data is mapped into vectors, queries are embedded in the same space, and nearby documents are supplied to the model so it can reason over current or application-specific information. He focuses on three unresolved engineering problems: choosing an embedding model, splitting data into useful chunks, and deciding whether retrieved results are relevant to a task or user. He recommends measuring retrieval with human feedback and existing information-retrieval benchmarks. For chunking, he discusses document structure, model-based semantic boundaries, hierarchical summaries, and embedding continuity. He also covers re-ranking, keyword search, metadata filtering, code retrieval, and querying structured data through natural-language summaries and SQL. Troynikov is candid that the field is still experimental. Chroma's design is aimed at applications with online updates and frequent user interactions, where he argues it differs from PGVector's typical use case.

## Key ideas
### Retrieval gives an LLM programmable memory
[03:39](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=219s)
Troynikov describes retrieval as a way to add context beyond the model's trained knowledge. An embedding model maps text, images, or video to a dense vector, which can be treated as a point on a map of meaning. Similar items tend to be close together. A typical system embeds a dataset, embeds the user's query, finds nearby documents, and passes those documents to the LLM. The model then spends more of its capacity drawing conclusions from supplied text rather than acting as a general-purpose knowledge store.

### Embedding systems still have several unanswered production questions
[07:48](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=468s)
Troynikov names three questions that developers repeatedly ask: which embedding model fits their data, how to divide documents into chunks, and whether returned results are actually relevant. Relevance can depend on the task and the user. Someone asking about a vehicle may care about speed, while another user cares about fuel economy. He says there are no complete answers yet. The technology is being adopted during a period of broad experimentation, while open-source tools and research are being used to develop better practices.

### Embedding model choice should be tested on the application's own data
[10:46](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=646s)
Different embedding models trained on the same data can produce maps with similar layouts, and Troynikov cites research suggesting that similar training objectives lead to similar representations. He recommends testing models rather than assuming that changing models will solve an application problem. BEIR, MTEB, and KILT provide information-retrieval benchmarks and evaluation frameworks. Teams can also collect task-specific labels through simple thumbs-up and thumbs-down feedback, embed that evaluation set with different models, and compare retrieval performance.

### Chunking must preserve context and the natural structure of the source
[12:44](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=764s)
Embedding models have finite context windows. If a document is too long, it may be truncated, leaving the embedding based only on its first part. Troynikov advises avoiding splits inside words or sentences because partial content can lose meaning and degrade retrieval. Documents often already have useful structure, such as chapters, sections, and paragraphs. NLTK can split text by sentences, sections, or paragraphs, while LangChain includes prebuilt chunking strategies.

### Experimental chunking methods can infer semantic boundaries
[14:50](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=890s)
Troynikov describes several approaches that go beyond fixed-size chunks. A language model can estimate next-token likelihood, with high perplexity potentially indicating a boundary between meanings. A hierarchical method can embed chapter summaries first, then search paragraph embeddings inside the selected chapter. Another approach measures continuity between successive embedding vectors and looks for unusually large distances. He presents these methods as experiments rather than established solutions.

### Retrieval quality improves when semantic search is combined with other signals
[16:56](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=1016s)
A retrieved neighbor is not automatically relevant to the query. Troynikov discusses re-ranking models, including one from Cohere, and using human feedback to tune ranking for a particular task or domain. Keyword search can run alongside semantic search, with a re-ranking model choosing among the results. Metadata can also narrow the search when the query identifies a documentation area or another known attribute. Chroma supports metadata filtering directly.

### Code retrieval should use code's existing semantic structure
[25:38](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=1538s)
For codebases, Troynikov recommends trying code-focused embedding models and benchmarking them against alternatives such as OpenAI's ada2. Code can be chunked by function, class, or file because those boundaries already carry meaning. A retrieval system can connect functions to their files and include documentation for a referenced function. This lets one query return both implementation and explanatory material. He also argues that keeping files and functions manageable makes this retrieval strategy easier.

### Structured data needs a bridge between natural language and database queries
[27:27](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=1647s)
Troynikov says embedding models do not currently handle tabular data directly very well. One approach is to ask a language model to summarize a table in natural language, embed that summary, and store a pointer to the table in metadata. The system can then retrieve the relevant table and use a language model to turn the natural-language request and table schema into SQL. Keyword search and direct database queries remain valid alongside vector search, depending on how the data is represented.

### Chroma is designed for changing application data
[22:46](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=1366s)
Troynikov contrasts Chroma with PGVector. He says PGVector works well for semantic search over a fixed dataset that changes infrequently, especially when an application already uses Postgres. He argues that larger datasets require more tuning and that online mutations can degrade performance. Chroma uses hierarchical navigable small world graphs, which he says are better suited to online updates and applications with frequent user interactions. He also says Chroma aims to package more of the retrieval capability so users do not need separate infrastructure and data-science specialists.

## Notable quotes
- "The basic idea is rather than just relying on the model's trained knowledge, we can actually add additional context to the model's input by pulling in relevant information relevant to our query." (03:39)
- "The bad news is today nobody really has the answers to these problems." (09:50)
- "You probably don't want to be dividing in the middle of sentences or even in the middle of words because you're losing meaning." (13:21)
- "Code is highly highly structured and there's no reason not to leverage that structure." (27:07)
- "All heuristics are brittle. If the heuristics worked you wouldn't need a model to actually be able to get the results that you want." (32:54)

## Tools & references mentioned
- Chroma
- Voyager
- Minecraft
- BEIR
- MTEB
- KILT
- NLTK
- LangChain
- Cohere
- PGVector
- OpenAI ada2
- Postgres
- SQL

## Who should watch
- You are building a retrieval-augmented generation system and need practical guidance on embedding model evaluation, chunking, and result ranking.
- Your application retrieves code, documentation, or structured database records and you need strategies that preserve the source's structure.
- You are comparing Chroma with PGVector for a system whose embedding data changes through ongoing user interactions.

## Related talks

- [Unlocking Real-World LLM Use Cases](https://mlopstalks.com/talks/unlocking-real-world-llm-use-cases) (Hamsa Buvaraghan, Google Cloud, 12:16)
- [Navigating via Retrieval Evaluation to Demystify LLM Wonderland](https://mlopstalks.com/talks/navigating-via-retrieval-evaluation-to-demystify-llm-wonderland) (Atita Arora, Qdrant, 12:53)
- [Evaluating and Integrating ML Models](https://mlopstalks.com/talks/evaluating-and-integrating-ml-models) (Morgan McGuire & Anish Shah, Weights & Biases, 51:57)
- [Solving LLM Data Problems](https://mlopstalks.com/talks/solving-llm-data-problems) (Yujian Tang, Zilliz, 28:32)
- [Democratizing AI](https://mlopstalks.com/talks/democratizing-ai) (Yujian Tang, Zilliz, 54:18)