Skylar Payne builds the system in small steps, starting with a terminal chat app, instrumentation, and a small evaluation set before adding retrieval.
2
The first LLM version answers 25% of the test queries, which shows that retrieval is not always needed when the model already knows enough.
3
A simple BM25s retriever over chunked documentation raises the evaluation result to 75%, while also exposing errors that need better documentation or more queries.
Summary
Skylar Payne builds a documentation chatbot live, starting with a hard-coded terminal chat app. He adds Lilyad tracing before changing the system, then collects four representative questions from the Miroscope Slack channel and manually annotates the results. The static bot scores 0%. Replacing it with an LLM raises the score to 25%, and Payne shows that a model can answer some library questions from its existing knowledge. He then downloads the full documentation, splits it into pages using content tags, indexes the pages with BM25s, and passes the retrieved text into the prompt. The resulting RAG system reaches 75% on the small evaluation set. Payne repeatedly favors the simplest implementation that can provide useful feedback. He also shows how tracing records function versions and nested retrieval and generation steps, which makes it easier to compare changes and find hallucinated answers.
The system starts with a deliberately minimal chat loop
Payne begins with a terminal application that reads user input and returns a hard-coded response, "Okay, how does that make you feel?" The bot is a simple function rather than a class, and the interface uses different colors for user and bot messages. It has no conversation state, which Payne accepts because the first goal is only to establish a working skeleton. He runs it with UV, checks the repeated response, and leaves small interface imperfections alone. The point is to get a runnable baseline before adding AI behavior.
Instrumentation comes before more complex AI behavior
Payne adds Lilyad tracing before building the RAG system because he wants visibility into every later change. Lilyad can version traced functions automatically, so changes to the bot response can be compared over time. He warns that teams can spend months implementing an AI system without evaluations, then fail to notice that the result is worse. His starting evaluation is intentionally lightweight. At this stage, a few queries can show large directional changes, such as moving from almost never working to sometimes working.
A small evaluation set is useful when its questions resemble real use
Payne collects questions from the Miroscope Slack channel rather than inventing easy test prompts. He stores the queries as Markdown files and gives each file an identifier. Four questions are enough to begin, even though he expects the set to grow. The questions include real technical details and have some complexity. He says a small set is not automatically useless, but it can mislead the project if its examples are not roughly representative of what users ask.
The hard-coded baseline makes evaluation behavior visible
The evaluation script loads each Markdown query, calls the bot response function, and records traces. Payne manually annotates the outputs in Lilyad as pass or fail. Since the bot always returns the same therapy-style sentence, it gets zero of four correct, or 0%. That poor result is expected. The exercise validates that the query-loading, tracing, annotation, and scoring workflow works before the application becomes more complicated.
An LLM can answer some questions without retrieval
Payne next replaces the static function with an LLM function and a prompt template. He provides basic information and an example program, then tests the model against the existing questions. The result reaches 25%. One answer about dynamically constructed text and image URLs is judged surprisingly good, while answers about Azure configuration, retry fallback, and max tokens contain errors. Payne uses this result to show that a full RAG system is not always the first required step. The model may already know enough for some queries.
Documentation is chunked as a separate preparation step
Payne downloads the combined documentation file and identifies page boundaries using content tags. He creates a separate script that takes the large Markdown file and writes one Markdown file per page, using the tag's title attribute for filenames. He keeps chunking outside the request path because documents that do not change often should not be reprocessed every time a user asks a question. He chooses this simple page-based split before trying more sophisticated chunking.
Payne indexes the generated documentation pages with BM25s, a lexical search implementation. Each Markdown file becomes a separate document, and the index is saved so it can be loaded later. At query time, the system tokenizes the user message, retrieves a small number of documents, and passes their text into the prompt. Payne starts with three retrieved documents without claiming that three is optimal. He intends to let the evaluation results guide changes to retrieval and chunking.
Tracing exposes retrieval and generation as separate steps
The RAG function keeps the original chat interface while adding a retrieval function and a response function that receives the retrieved documents. Lilyad records the calls as nested steps, with document retrieval under the main response and generation after it. Payne uses a debugger when the retriever returns an unexpected NumPy structure, inspects the result shape and fields, and extracts the document text. This is part of his broader method: inspect the smallest failing piece, fix it, and rerun the application.
The first RAG version improves the score, then exposes the limits of the test set
After adding retrieved documentation, Payne annotates the same questions again. The score rises from 0% for the hard-coded bot, to 25% for the standalone LLM, and then to 75% with retrieval. Some answers still use the wrong Azure setup or invent retry details, while other answers correctly use documented settings such as call parameters for max tokens. Payne says the four-query set is now saturated, so the next step is to collect more examples and build a more automated evaluation loop.
