# Small Models, Big Ideas: The Next Frontier in AI

Korri Jones, Chick-fil-A Corporate Support Center & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Sophia Skowronski, Breckinridge Capital Advisors & Lihu Chen, Imperial College London & Binoy Perera, MLOps Community | MLOps Reading Group | 58:44

Source: https://www.youtube.com/watch?v=FjXe23BM3DA
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/small-models-big-ideas-the-next-frontier-in-ai
Published: 2024-11-05
Tags: distillation, inference, rag, synthetic-data

## TL;DR
- Small language models remain useful because they need fewer computing resources and can perform well on specific tasks.
- Small models can work with large models by filtering data, evaluating outputs, routing requests, improving prompts, and supplying missing capabilities.
- Model compression and synthetic data can make language models cheaper to run, but smaller models usually give up some generalization.

## Summary
The paper "What is the Role of Small Models in the LLM Era: A Survey" reviews why small language models still matter alongside large language models. Lihu Chen explains that limited GPU access motivated some of his own work, including using a small model to help an encoder handle out-of-vocabulary words. The paper groups small-model uses into collaboration and competition. Small models can curate training data, supervise larger models, evaluate outputs, route requests, retrieve information for prompts, and help estimate uncertainty. They can also be compressed through knowledge distillation or trained with synthetic data generated by a larger model. The discussion covers edge devices, domain-specific tasks, interpretability, retrieval-augmented generation, weak-to-strong generalization, model cascading, model routing, contrastive decoding, and prompt selection. The presenters are clear that larger models usually generalize better, while small models can be cheaper, faster, and better suited to narrow tasks.

## Key ideas
### Limited compute can make small-model research the practical choice
[04:57](https://www.youtube.com/watch?v=FjXe23BM3DA&t=297s)
Lihu Chen says his first motivation was simple: he did not have many GPUs or the computing resources available to industrial labs. During his PhD, pretraining a BERT-based model from scratch took more than a month on the lab's available hardware, and repeating experiments became difficult. Reviewers also asked him to study much larger models such as Llama 7B. Chen took this as evidence that researchers with fewer resources cannot compete by copying the same large-model experiments. Small models let researchers test ideas with a manageable training cost.

### A small model can repair a specific weakness in a larger model
[08:00](https://www.youtube.com/watch?v=FjXe23BM3DA&t=480s)
Chen describes work on out-of-vocabulary words in encoder language models. An encoder can make mistakes when it sees a word that was absent from pretraining. His proposed approach used a small model as a plug-in rather than retraining the original model from scratch. The smaller model learned to provide information for unfamiliar words, and the two models worked together. Chen presents this as an early example of small models enhancing larger models on a narrow problem. The group also discusses small models as evaluators that can check consistency, confidence, or quality in generated responses.

### Small models have practical advantages on constrained or specialized workloads
[12:04](https://www.youtube.com/watch?v=FjXe23BM3DA&t=724s)
Chen says large language models generally achieve stronger results and transfer better to unseen tasks, but that does not make them the right choice for every deployment. Small models can fit settings with limited computing resources, including phones and edge devices. He also points to healthcare and legal question answering, where users may need a model whose behavior is easier to inspect. For domain-specific work, tabular learning, and short-text reasoning, a small model trained on limited data can outperform a general-purpose large model. The benefit comes from matching model size to the task.

### Small models can collaborate with large models during inference
[16:17](https://www.youtube.com/watch?v=FjXe23BM3DA&t=977s)
The paper describes several ways for models of different sizes to work together. A small model can estimate the confidence or consistency of a large model's answer. A cascade can send an input to a smaller model first and pass difficult cases to a more capable model. A routing model can select the best model before running every candidate. The presenters compare this with a support chain in which simple questions are handled early and harder cases are escalated. These approaches can reduce inference speed and financial costs when many requests do not require the largest model.

### Carefully selected data can reduce the need for larger training runs
[27:56](https://www.youtube.com/watch?v=FjXe23BM3DA&t=1676s)
Valdimar Eggertsson discusses the survey's section on data curation. Training on the entire internet gives a model a great deal of material, but the data also contains low-quality content. The paper describes using smaller models or classifiers to filter examples, assess content quality, and reweight sources. A paragraph from a scientific paper might receive more weight than a tweet. The discussion also mentions work suggesting that carefully selected textbooks or instruction examples can produce capable models with less data. The group raises a limitation: removing toxic material may also reduce a model's ability to recognize or respond to toxic language.

### Weak models can help align stronger models
[35:08](https://www.youtube.com/watch?v=FjXe23BM3DA&t=2108s)
The weak-to-strong generalization idea reverses the traditional direction of knowledge transfer. Instead of using a stronger model to supervise a smaller one, a smaller model supplies labels or supervision to a stronger model. Chen explains that this may matter when a future model produces code or other outputs that are too long and complicated for humans to evaluate reliably. A weaker model can provide an automated signal for alignment. The paper's claim is about alignment rather than general performance, and the discussion stresses that a small model here means one that is smaller than the model it supervises.

### Retrieval and prompt selection are useful places for small models
[45:24](https://www.youtube.com/watch?v=FjXe23BM3DA&t=2724s)
Sophia Skowronski explains how the paper applies small models to retrieval-augmented generation and prompt-based learning. Sparse retrievers can use methods such as term frequency and BM25, while dense retrievers can use an encoder such as BERT to compare representations. A small model can retrieve documents, select prompts, or break a complex prompt into smaller subproblems before a large model answers. The examples include systems that generate search queries, retrieve knowledge from vector stores or structured sources, and add the retrieved material to the large model's context. This keeps the large model focused on the final generation step.

### Compression transfers a large model's behavior into a smaller one
[53:20](https://www.youtube.com/watch?v=FjXe23BM3DA&t=3200s)
The discussion distinguishes contrastive decoding from knowledge distillation. Contrastive decoding combines the outputs of a larger expert model and a smaller amateur model during inference, choosing tokens where the expert is more confident. Knowledge distillation happens during training. The student learns to imitate a teacher's outputs, sometimes with access to the teacher's internal architecture and sometimes using only its outputs. DistilBERT is given as an example of white-box distillation. Black-box approaches can transfer reasoning chains or instruction-following behavior without access to the teacher's internal parameters.

## Notable quotes
- Lihu Chen: "I stick to small models or emphasize small-scale language models because I don't have very many GPUs." (05:19)
- Lihu Chen: "For some certain tasks we don't need larger language models, we can still use small ones." (12:48)
- Lihu Chen: "In general larger models are more generalized compared to smaller ones." (23:03)
- Valdimar Eggertsson: "We can use smaller models when we can, and we don't need to always just use the sledgehammer when you can use a hammer." (38:49)

## Tools & references mentioned
- What is the Role of Small Models in the LLM Era: A Survey
- Llama 7B
- BERT
- RoBERTa
- Llama
- OpenAI
- Weak-to-Strong Generalization
- Hugging Face
- GPT-3
- GPT-4
- GPT-5
- BM25
- DistilBERT
- Retrieval-Augmented Generation
- Knowledge GPT
- UPRISE
- SLM
- Chain of Thought
- contrastive decoding

## Who should watch
- You are deciding whether every language task needs a large model and want concrete cases where a smaller model may be enough.
- You are building an inference system and need ideas for routing, cascading, confidence checks, or retrieval before calling a large model.
- You are studying model compression, distillation, alignment, or small-model support for edge and domain-specific applications.

## Related talks

- [Small Language Models are the Future of Agentic AI](https://mlopstalks.com/talks/small-language-models-are-the-future-of-agentic-ai) (Adam Becker, MLOps Community & Nehil Jain, Stealth AI Startup & Sonam Gupta, AI Camp, 58:13)
- [RAG: The Future of AI Search (and Prompt Engineering)](https://mlopstalks.com/talks/rag-the-future-of-ai-search-and-prompt-engineering) (Korri Jones, Chick-fil-A Corporate Support Center & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Sophia Skowronski, Breckinridge Capital Advisors & Lihu Chen & Binoy Perera, 10:13)
- [Language, Graphs, and AI in Industry](https://mlopstalks.com/talks/language-graphs-and-ai-in-industry) (Paco Nathan, Derwen, Inc., 1:18:29)
- [What Drives GenAI Development in the Next 3 Years](https://mlopstalks.com/talks/what-drives-genai-development-in-the-next-3-years) (Euro Beinat, Prosus, 18:54)
- [Fine-Tuning LLMs: Best Practices and When to Go Small](https://mlopstalks.com/talks/fine-tuning-llms-best-practices-and-when-to-go-small) (Mark Kim-Huang, Preemo Inc., 53:48)