# Finetuning Open-Source LLMs

Sebastian Raschka, Lightning AI | LLMs in Production 2023 | 29:04

Source: https://www.youtube.com/watch?v=IYuW2Ez4V2M
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/finetuning-open-source-llms
Published: 2023-10-18
Tags: cost, fine-tuning, open-models

## TL;DR
- Prompting and retrieval-augmented generation are useful ways to work with pretrained language models, but fine-tuning adapts a model to a custom task.
- LoRA factorizes weight updates into smaller matrices, which can reduce the resources needed to fine-tune a large model.
- Full fine-tuning can be a useful baseline, while updating only selected layers or using LoRA can give a better fit for limited budgets and hardware.

## Summary
Sebastian Raschka explains how pretrained language models can be used before discussing fine-tuning. Prompting changes the model's input, while retrieval-augmented generation adds information from a document collection. Fine-tuning changes the model for a target task, including classification and instruction following. He compares feature extraction, updating output layers, and updating the full Transformer. Full updates can improve performance, but they use more time, memory, and compute. Raschka then explains low-rank adaptation, or LoRA, which represents a weight update with two smaller matrices. In his examples, LoRA reduced a 7-billion-parameter training run from about nine hours on six GPUs to about one hour on one GPU. He also demonstrates an open-source Lightning AI repository for downloading models, preparing datasets, and running LoRA or full fine-tuning. His advice is practical: use full fine-tuning when resources allow, then test whether fewer layers or LoRA provide enough performance for the task.

## Key ideas
### Pretrained models support several workflows before fine-tuning
[02:29](https://www.youtube.com/watch?v=IYuW2Ez4V2M&t=149s)
Raschka starts with prompting, where the wording and format of the input can change the result, especially for smaller open-source models. He then describes retrieval-augmented generation as a hybrid system. A company can split documentation into chunks, create embeddings, store them in a vector database, and retrieve similar passages for a user query. The language model uses that retrieved material when generating its answer. This is useful when the needed information already exists in company documentation and should not be regenerated from memory.

### Classification is a practical reason to fine-tune a language model
[06:32](https://www.youtube.com/watch?v=IYuW2Ez4V2M&t=392s)
Raschka argues that many real applications involve classification, including spam detection, fake-news detection, and toxic-content detection. One option is to extract embeddings from a frozen Transformer and train a separate classifier such as logistic regression or a support vector machine. A second option adds and fine-tunes output layers while keeping the Transformer frozen. A third option updates the entire language model. In his movie-review example with 50,000 examples, the small model reached about 90% accuracy after three minutes when only some parts were updated, and 93% when the whole model was updated.

### Instruction fine-tuning teaches a model to produce task-specific responses
[10:13](https://www.youtube.com/watch?v=IYuW2Ez4V2M&t=613s)
Instruction fine-tuning uses examples containing an instruction, an optional input, and the desired output. Raschka gives examples such as asking for a limerick about a pelican or identifying the odd item in a group. This differs from pretraining, where the model predicts the next word in unlabelled text. He compares a GPT-3 base model, the same model with prompting, and a model with supervised fine-tuning. The supervised version performs better in the comparison, which is why many high-performing models on leaderboards use supervised fine-tuning.

### LoRA reduces the number of trainable parameters
[12:29](https://www.youtube.com/watch?v=IYuW2Ez4V2M&t=749s)
Low-rank adaptation, or LoRA, represents the weight update as two smaller matrices rather than storing a full update matrix. Raschka gives a 5,000 by 1,000 matrix as an example. The full matrix has 5 million parameters, while factorizing it into matrices sized 5,000 by 10 and 10 by 1,000 uses 60,000 parameters. LoRA can approach full fine-tuning performance for particular tasks, although Raschka does not present it as a general guarantee. The rank and the layers selected for updating are important hyperparameters.

### The resource difference between full fine-tuning and LoRA is large
[14:28](https://www.youtube.com/watch?v=IYuW2Ez4V2M&t=868s)
In Raschka's speed test, full fine-tuning of a 7-billion-parameter model took about nine hours on six GPUs. He needed CPU offloading because the full model did not fit across those GPUs. LoRA fit on one GPU and took about one hour for 50,000 training examples. Full fine-tuning used the available GPU memory, while LoRA used about 16 GB. Quantized LoRA reduced memory use further, to roughly 12 or 13 GB, although the extra quantization step added computation time.

### An open-source training repository keeps the process inspectable
[16:15](https://www.youtube.com/watch?v=IYuW2Ez4V2M&t=975s)
Raschka describes an open-source Lightning AI repository that provides scripts for fine-tuning models such as Mistral and Llama 2. He prefers it because users can inspect and modify the scripts instead of working through a fully hidden interface. The basic workflow is to clone the repository, install its requirements, download a model, convert its checkpoint, prepare a dataset, and run a training command. The repository supports built-in datasets and custom CSV files. Dataset preparation must match the model's tokenizer, since different models can tokenize the same text differently.

### A small number of Transformer blocks may be enough
[24:38](https://www.youtube.com/watch?v=IYuW2Ez4V2M&t=1478s)
When asked about full fine-tuning, Raschka describes experiments that started with the output layer and added Transformer blocks one at a time. In the experiment he discusses, performance had already saturated after updating roughly two or three Transformer blocks, even though the model had more blocks available. He recommends starting with the output layer and gradually adding layers to see whether the extra updates are useful. The result depends on the dataset, so this is a testing strategy rather than a fixed rule.

### The fine-tuning choice depends on budget and the target task
[25:54](https://www.youtube.com/watch?v=IYuW2Ez4V2M&t=1554s)
Raschka calls full fine-tuning a useful baseline when enough resources are available, but the cost can become substantial. He estimates that eight expensive GPUs can cost around 30 or 40 dollars per hour, so a ten-hour experiment can cost about 400 dollars. LoRA can provide a good approximation at lower cost, and he suggests that a team might prefer LoRA on a model ten times larger rather than full fine-tuning on a smaller model. He also says LoRA can reduce performance on tasks that were not part of the fine-tuning target, which may be acceptable when the application has a narrow goal.

## Notable quotes
- "The focus of this talk is on fine-tuning where we take a pre-trained large language model and then fine-tune it on custom target tasks." (06:11)
- "If I had to pick one specific technique then that would be low rank adaptation LoRA, which is currently the most popular one." (12:29)
- "With LoRA I could fit that on one GPU and train that in about one hour." (15:08)
- "It is really to make it better on certain tasks you care about in a certain application scenario." (23:14)

## Tools & references mentioned
- Lightning AI
- ChatGPT
- Mathpix
- GPT-3
- LoRA
- retrieval-augmented generation
- support vector machine
- logistic regression
- recurrent neural network
- Llama 2
- Falcon
- LongChat
- StableLM
- Code Llama
- Mistral
- Mistral 7B
- Alpaca
- NeurIPS Large Language Model Efficiency Challenge
- bf16
- NF4

## Who should watch
- You are choosing between prompting, retrieval-augmented generation, and fine-tuning for an application with a defined task.
- You need to adapt an open-source language model but have limited GPU memory or a limited training budget.
- You want a practical starting point for comparing full fine-tuning, selected-layer updates, and LoRA.

## Related talks

- [Fine-Tuning LLMs: Best Practices and When to Go Small](https://mlopstalks.com/talks/fine-tuning-llms-best-practices-and-when-to-go-small) (Mark Kim-Huang, Preemo Inc., 53:48)
- [From Research to Production: Fine-Tuning & Aligning LLMs](https://mlopstalks.com/talks/from-research-to-production-fine-tuning-aligning-llms) (Philipp Schmid, Hugging Face, 38:03)
- [Tricks to Fine Tuning](https://mlopstalks.com/talks/tricks-to-fine-tuning) (Prithviraj Ammanabrolu, Databricks, 54:02)
- [Fine Tuning Llamas](https://mlopstalks.com/talks/fine-tuning-llamas) (Kai Davenport, 13:37)
- [Understanding the LLM Economics](https://mlopstalks.com/talks/understanding-the-llm-economics) (Nikunj Bajaj, TrueFoundry, 31:23)