The paper groups retrieval-augmented generation techniques around textual documents, structured knowledge, and retrieval systems that support language models.
2
Small models can retrieve prompts, break complex prompts into smaller problems, or improve large-model outputs during inference.
3
Knowledge distillation transfers behavior from a larger model into a smaller one, while synthetic data gives small models more examples when human-labeled data is limited.
Summary
What is the Role of Small Models in the LLM Era: A Survey is the paper discussed in this reading group. Sophia Skowronski uses retrieval-augmented generation as an entry point for explaining how small models can support large language models. She covers sparse retrieval with TF-IDF and BM25, dense retrieval with BERT-style encoders, and systems that combine vector databases with entity trees or other structured knowledge. Small models can also select prompts, decompose complex prompts, and improve token selection through contrastive decoding. The session distinguishes inference-time methods from training methods. Contrastive decoding compares a larger expert model with a smaller amateur model, while knowledge distillation trains a student model to reproduce a teacher's behavior. Sophia also discusses white-box and black-box distillation, including DistilBERT, chain-of-thought transfer, instruction following, and synthetic data generation. She is careful about synthetic data in regulated fields, where altered examples could change information that needs to remain precise.
RAG adds external knowledge to a language model's context
Sophia Skowronski describes retrieval-augmented generation as a way to combine a large language model with an external knowledge base. The system retrieves relevant information and places it in the model's context. This addresses problems that Lihu Chen discusses around memorization and hallucination. The paper divides retrieval approaches into areas that include textual documents and structured knowledge. Sophia presents RAG as a setting where small models can handle retrieval while a larger model produces the final response.
Sparse and dense retrievers match queries to documents in different ways
For textual documents, Sophia contrasts sparse and dense retrieval. Sparse retrievers use vocabulary and term-based relevance methods such as term frequency-inverse document frequency and BM25, also called Best Match 25. She says these methods require specific vocabulary to be present and may not generalize well. Dense retrieval uses an encoder such as BERT for queries and documents, then compares their representations for similarity. The slide examples show that smaller fine-tuned language models can take part in the retrieval process.
Knowledge GPT and T-RAG use extra structure around retrieval
Sophia explains that Knowledge GPT uses a program-of-thought prompting technique to generate search queries for knowledge bases. It separates knowledge retrieval from knowledge storage, generates search code, executes that code, and sends the retrieved knowledge to a language model. She then describes T-RAG as a system that searches a vector database and adds information from an entity tree when an organization is mentioned. These systems use structured knowledge from knowledge graphs, tables, or databases alongside retrieved document chunks.
Small models can improve prompt selection and prompt decomposition
Prompt-based learning guides a frozen language model without changing its parameters, while in-context learning places examples inside the prompt. Sophia describes UPRISE as a lightweight retriever that selects prompts from a prepared pool, adds one to an input, and uses the frozen language model's evaluation to train the retriever. She also describes SLAM, which uses a generator-retriever to break a complex prompt into subproblems that need fewer reasoning steps. A larger solver language model then answers the expanded input.
Contrastive decoding changes inference without training a new model
Deep contrastive decoding compares a larger, more capable expert model with a smaller amateur model. The method chooses tokens based on the difference in their log likelihoods, favoring tokens that the expert finds more probable. Sophia stresses that this happens during inference. It does not transfer knowledge or update model parameters. The models' combined outputs determine the next token.
Knowledge distillation compresses a larger model into a smaller one
Sophia distinguishes contrastive decoding from knowledge distillation, which is a training method. In white-box distillation, the student has access to the teacher's internal architecture and parameters. She uses DistilBERT as an example of this approach, with the goal of compressing the model and improving efficiency. In black-box distillation, only the teacher's outputs are available, so the student learns to produce similar outputs for the same inputs. The paper's examples include transferring chain-of-thought reasoning and instruction-following behavior.
Synthetic data expands training material but can be risky in regulated settings
Large language models can generate synthetic data for small models when human-graded data is limited. Sophia separates data generation from data augmentation. Augmentation can ask a model to create alternative examples from the same base text. She warns that this approach may be unsuitable for heavily regulated areas such as medicine and finance. In those settings, changing generated examples could alter information tied to a specific standard of care or other precise requirements.
"RAG is a technique that enhances large language models by combining them with an external knowledge base and then retrieves relevant information and embeds it in the context given to a language model."Sophia Skowronski00:00
Who should watch
You are building a RAG system and need a compact explanation of sparse retrieval, dense retrieval, vector databases, and structured knowledge.
You use large-model APIs and want to see where small models can help with prompt selection, prompt decomposition, or inference-time decoding.
You are evaluating model compression or synthetic-data methods and need to understand the difference between white-box distillation, black-box distillation, and generated training examples.
