Retrieval gives an LLM programmable memory by finding documents near a query in embedding space and adding them to the model's context.
2
Embedding model choice, document chunking, and retrieval relevance still require application-specific testing because there are no settled answers.
3
Good retrieval systems combine semantic search with document structure, metadata filters, keyword search, re-ranking, and feedback from users.
Summary
Anton Troynikov explains how embeddings let an application add external context to an LLM. Data is mapped into vectors, queries are embedded in the same space, and nearby documents are supplied to the model so it can reason over current or application-specific information. He focuses on three unresolved engineering problems: choosing an embedding model, splitting data into useful chunks, and deciding whether retrieved results are relevant to a task or user. He recommends measuring retrieval with human feedback and existing information-retrieval benchmarks. For chunking, he discusses document structure, model-based semantic boundaries, hierarchical summaries, and embedding continuity. He also covers re-ranking, keyword search, metadata filtering, code retrieval, and querying structured data through natural-language summaries and SQL. Troynikov is candid that the field is still experimental. Chroma's design is aimed at applications with online updates and frequent user interactions, where he argues it differs from PGVector's typical use case.
Troynikov describes retrieval as a way to add context beyond the model's trained knowledge. An embedding model maps text, images, or video to a dense vector, which can be treated as a point on a map of meaning. Similar items tend to be close together. A typical system embeds a dataset, embeds the user's query, finds nearby documents, and passes those documents to the LLM. The model then spends more of its capacity drawing conclusions from supplied text rather than acting as a general-purpose knowledge store.
Embedding systems still have several unanswered production questions
Troynikov names three questions that developers repeatedly ask: which embedding model fits their data, how to divide documents into chunks, and whether returned results are actually relevant. Relevance can depend on the task and the user. Someone asking about a vehicle may care about speed, while another user cares about fuel economy. He says there are no complete answers yet. The technology is being adopted during a period of broad experimentation, while open-source tools and research are being used to develop better practices.
Embedding model choice should be tested on the application's own data
Different embedding models trained on the same data can produce maps with similar layouts, and Troynikov cites research suggesting that similar training objectives lead to similar representations. He recommends testing models rather than assuming that changing models will solve an application problem. BEIR, MTEB, and KILT provide information-retrieval benchmarks and evaluation frameworks. Teams can also collect task-specific labels through simple thumbs-up and thumbs-down feedback, embed that evaluation set with different models, and compare retrieval performance.
Chunking must preserve context and the natural structure of the source
Embedding models have finite context windows. If a document is too long, it may be truncated, leaving the embedding based only on its first part. Troynikov advises avoiding splits inside words or sentences because partial content can lose meaning and degrade retrieval. Documents often already have useful structure, such as chapters, sections, and paragraphs. NLTK can split text by sentences, sections, or paragraphs, while LangChain includes prebuilt chunking strategies.
Experimental chunking methods can infer semantic boundaries
Troynikov describes several approaches that go beyond fixed-size chunks. A language model can estimate next-token likelihood, with high perplexity potentially indicating a boundary between meanings. A hierarchical method can embed chapter summaries first, then search paragraph embeddings inside the selected chapter. Another approach measures continuity between successive embedding vectors and looks for unusually large distances. He presents these methods as experiments rather than established solutions.
Retrieval quality improves when semantic search is combined with other signals
A retrieved neighbor is not automatically relevant to the query. Troynikov discusses re-ranking models, including one from Cohere, and using human feedback to tune ranking for a particular task or domain. Keyword search can run alongside semantic search, with a re-ranking model choosing among the results. Metadata can also narrow the search when the query identifies a documentation area or another known attribute. Chroma supports metadata filtering directly.
Code retrieval should use code's existing semantic structure
For codebases, Troynikov recommends trying code-focused embedding models and benchmarking them against alternatives such as OpenAI's ada2. Code can be chunked by function, class, or file because those boundaries already carry meaning. A retrieval system can connect functions to their files and include documentation for a referenced function. This lets one query return both implementation and explanatory material. He also argues that keeping files and functions manageable makes this retrieval strategy easier.
Structured data needs a bridge between natural language and database queries
Troynikov says embedding models do not currently handle tabular data directly very well. One approach is to ask a language model to summarize a table in natural language, embed that summary, and store a pointer to the table in metadata. The system can then retrieve the relevant table and use a language model to turn the natural-language request and table schema into SQL. Keyword search and direct database queries remain valid alongside vector search, depending on how the data is represented.
Troynikov contrasts Chroma with PGVector. He says PGVector works well for semantic search over a fixed dataset that changes infrequently, especially when an application already uses Postgres. He argues that larger datasets require more tuning and that online mutations can degrade performance. Chroma uses hierarchical navigable small world graphs, which he says are better suited to online updates and applications with frequent user interactions. He also says Chroma aims to package more of the retrieval capability so users do not need separate infrastructure and data-science specialists.
"The basic idea is rather than just relying on the model's trained knowledge, we can actually add additional context to the model's input by pulling in relevant information relevant to our query."03:39
Who should watch
You are building a retrieval-augmented generation system and need practical guidance on embedding model evaluation, chunking, and result ranking.
Your application retrieves code, documentation, or structured database records and you need strategies that preserve the source's structure.
You are comparing Chroma with PGVector for a system whose embedding data changes through ongoing user interactions.
