Wes Ladd found that unreliable source files and poorly structured websites made data preparation one of the hardest parts of building the system.
2
Search quality depends on tradeoffs between recall, latency, database control, and the indexing algorithm used.
3
Embedding model choice and context chunking depend on the data, the evaluation metric, and how well the chunks preserve meaning.
Summary
Wes Ladd describes the problems his team faced while building a retrieval augmented generation system for StinkBait, a cybersecurity research and reporting application from Train GRC. Cybersecurity information is fragmented across communities, and language models may censor some security topics, so the team needed its own retrieval system. Ladd focuses on the work before generation: validating PDFs, extracting text, scraping websites, choosing search indexes, selecting embedding models, and chunking context. Invalid PDFs may need to be converted to JPEG before Amazon Textract can process them. Website extraction often requires manual inspection because HTML structure varies. Search indexes involve recall and latency tradeoffs, while embedding models differ by context length, evaluation results, language, modality, and training data. Ladd is also direct about the limits of heading-based chunking. Formatting can approximate topic boundaries, but it does not guarantee semantic separation.
A RAG system may need its own cybersecurity knowledge base
Train GRC built StinkBait, an application for cybersecurity research and reporting. Wes Ladd says cybersecurity knowledge is fragmented across different subcommunities, which makes collecting a useful knowledge base difficult. Security topics can also trigger refusals from general language models, such as a response that the model cannot help for ethical or safety reasons. The team therefore decided to build a retrieval augmented generation system that could work with this information directly. The motivation was practical: gather dispersed material and provide relevant context for cybersecurity research rather than rely only on a general model's built-in knowledge.
Ladd describes a team question: "is it really a PDF?" Files labelled as PDFs can come from different extensions, website plugins, or poor photocopies, and some are not valid PDFs at all. This matters because the extraction path depends on the file being processed correctly. The team uses Amazon Textract for optical character recognition, but Textract can report that an input is not actually a PDF. Their fallback is to transform the file into a JPEG and process the image instead. The example shows why ingestion needs validation and recovery paths before documents enter a retrieval system.
Web content creates a separate ingestion problem. Websites use different frameworks and what-you-see-is-what-you-get editors, so their HTML is not consistently structured. Ladd says that producing high-quality data can require manually inspecting sites and identifying which HTML elements contain the useful material. Open-source and third-party scraping tools may return data that looks less curated than expected. The team therefore treats extraction quality as a site-specific task. A scraper that runs successfully can still produce unsuitable retrieval data if it captures navigation, unrelated page elements, or poorly organised content.
Vector search indexes trade recall against latency
Ladd distinguishes exact k-nearest-neighbour search from approximate nearest-neighbour search. Exact search can have too much latency for a real-time application, so teams may use indexes such as IVF flat or HNSW. Each index has different tradeoffs between recall or accuracy and request latency. The team initially considered PostgreSQL with pgvector, partly because an open-source database could provide flexibility as the organisation changed. At the time, pgvector supported IVF flat, and the latency results made them reconsider that choice. A vector database may hide some algorithm decisions, while a self-managed approach gives more control and more operational responsibility.
Ladd says many teams choose Ada embeddings because they already use the OpenAI API. He points out that Ada embeddings has a long context length and can take almost five pages of a document in one embedding. It also performs well on the Hugging Face MTE leaderboard, but it is not the best model for every evaluation metric or use case. Other choices include Sentence Transformers trained for short question-and-answer pairs, MS MARCO passage models for longer text, multilingual models, and multimodal models that handle images and text. The model's training data and the form of the source material both affect the choice.
The ideal chunk would follow a conceptual boundary, with information about one topic kept together and a different topic placed in another embedding. Ladd says that this is difficult to achieve at scale. A more realistic approach is to use formatting, such as Markdown heading levels, to split text. Headings provide a proxy for contextual separation, but they do not guarantee that the resulting sections are semantically complete. When the system injects retrieved context into generation, the relevance of that context depends partly on how it was chunked. A formatting boundary can be useful while still cutting across the meaning of the source.
Store vectors outside the database when experimentation is likely
Changing vector databases or embedding models can force expensive recomputation if vectors exist only inside the current system. Ladd suggests storing vectors on disk or in blob storage while experimenting with vector databases, so they can be loaded into another database later. Embedding experiments also require decisions about vector dimensions and the storage space those dimensions will consume. The source data's style matters as well. Choosing a model that does not fit the data can prevent the system from producing the desired retrieval results, so storage and model decisions need attention before the system is locked in.
