A production RAG application needs careful document chunking, retrieval, evaluation, and model selection.
2
A dataset of about 200 annotated community questions lets the team test retrieval, answer generation, and the complete system.
3
Ray Data and Ray Serve help scale embedding pipelines and deploy low-latency, composable model services.
Summary
Philipp Moritz and Yifei Feng describe a question-answering assistant for Ray documentation, blogs, and books. They explain the RAG pipeline: load source documents, split them into meaningful chunks, embed the chunks, retrieve relevant passages for each query, and pass them to a language model. The team evaluated about 200 community questions with annotated source sections, using both component tests and end-to-end tests. They also used an LLM judge to score answers on a one-to-five scale. Their experiments covered chunk size, the number of retrieved chunks, embedding models, and language models. For production, Ray Data scales document processing and embedding, while Ray Serve handles model composition, streaming, autoscaling, and deployment changes. The talk also covers feedback-driven improvement, reranking, fine-tuning, and routing simple questions to cheaper models. Anyscale Endpoints provides an OpenAI-compatible way to serve open-source models without hosting them directly.
RAG supplies the private and changing context that general-purpose language models lack
Yifei Feng says a standalone language model is a poor fit for the Ray assistant because Ray documentation changes as new features are released, and a general-purpose model does not contain private data or intellectual property. Hallucination is another concern because the assistant must not invent code. RAG addresses this by finding relevant passages in the documentation and placing them into the model's context. One example question about ray.wait requires information about both ray.wait itself and Ray's fault-tolerance behavior. The system must retrieve both pieces and combine them into one answer.
Document chunking must preserve the structure and meaning of the source
The pipeline starts with Ray documentation, blogs, and books. The team loads the material, breaks it into smaller chunks, embeds each chunk, and stores the vectors with the text and its source. Philipp Moritz says chunking depends on the data. For HTML, they split by section and retain the section anchor. Long sections are split into paragraphs or code snippets, while code is kept together where possible. The reason is practical: embedding models work better on focused passages, and cutting code in the middle can confuse the language model and reduce answer quality.
A small annotated question set supports both component tests and end-to-end tests
The team built a dataset of about 200 questions from the Ray community and annotated each question with the documentation chunks that answer it. This supports component-level checks, similar to unit tests. Retrieval can be tested by checking whether the annotated passages appear among the results. The language-model step can be tested with fixed ground-truth context. End-to-end tests run the whole system from query to final response. This gives the team a repeatable way to check whether later changes to chunking, retrieval, prompts, or models have made the application worse.
LLM-based grading makes repeated evaluation more practical
Human review is useful but does not scale for every experiment. The team instead gives an evaluation model the question, context, and proposed answer, then asks it to assign a score from one to five and explain the score. Philipp Moritz says this does not replace human evaluation, but it works well enough to run automatically after system changes. The same evaluation set can compare retrieval quality with answer quality. The team reported an overall score of 3.6 in its experiments, using these scores to compare configurations rather than treating a single model choice as universally best.
More context can improve answers while making retrieval less precise
The experiments varied the number of retrieved chunks from one to seven and chunk sizes from 100 to 700 words. Adding context performed better than giving the model no retrieved context. Larger chunks generally improved the final answer because they contained more information, but retrieval scores fell as chunks became larger. The embedding model had more difficulty representing all the information in a large chunk. Increasing the number of chunks also often helped, until the language model's context limit became the constraint. These results show why chunk size and retrieval count need to be tested together.
The best embedding model depends on the application, not only on a public leaderboard
The team compared embedding models from the Hugging Face leaderboard with OpenAI embeddings. The model that ranked highest on the public leaderboard, BGE large, was not the best choice for this Ray documentation application. Another open-source model, GTE base, performed better in their tests. OpenAI embeddings scored slightly below the best open-source model. Among language models, GPT-4 had the highest score at 3.8, while GPT-3.5 Turbo cost less and performed well. Llama 70B was close to GPT-3.5 Turbo, while Falcon 180B performed slightly worse than Falcon 70B in this evaluation.
Production RAG needs separate scaling for data preparation and model serving
Embedding a large document collection and running evaluations require substantial computation, including GPU work. Ray Data handles multiple input types and sources, supports CPU and GPU processing, and lets the team scale those resources independently. Serving is a separate problem because the application combines embedding, retrieval, ranking, and language-model components. Ray Serve supports this composition and provides end-to-end streaming, so generated tokens can start reaching the user before the complete answer is finished. The deployment also needs zero-downtime upgrades, canary rollouts, rollback, request metrics, user feedback, autoscaling, and flexible resource allocation.
User feedback can drive documentation changes, reranking, and model retraining
After deployment, the team can collect the questions users ask, the chunks returned by the vector database, and whether users find the response helpful. That feedback can lead to better documentation, new chunking choices, or another run of the data pipeline. The embedding model can also be fine-tuned with Ray-specific knowledge. Reranking provides another way to improve retrieval by selecting the most relevant documents from an initial group of results. Yifei Feng presents Ray Train as a way to repeat this work across familiar deep-learning frameworks and different CPU, GPU, or custom hardware configurations.
Routing simple questions to cheaper models can reduce inference cost
The language-model stage is the most computationally expensive part of the application. Yifei Feng says GPT-4 performed better in their comparison but cost 47 times as much as Llama 70B. Some questions only need a response copied from the documentation, while a code-example request may work with a smaller open-source model. The team used Ray Serve to compose a classifier with several models and route questions according to their needs. Each model can autoscale separately, so the system does not pay for unused capacity. Anyscale Endpoints offers an OpenAI-compatible way to use open-source models without managing the serving infrastructure directly.
"This is a good example of how such system can find these two different piece of information put it together in a reasonable way and present that to the user."Yifei Feng03:53
Who should watch
You are building a documentation assistant and need a concrete RAG design with retrieval and answer evaluation.
Your team needs to scale document processing, embeddings, or model serving across CPUs and GPUs.
You are comparing model quality with inference cost and want to route different questions to different models.
