Fine-tuning adapts an open LLM to a task with far less compute and data than pre-training, often using instruction examples and parameter-efficient methods.
2
Good instruction data needs clear directions, varied tasks, consistent formatting, and human review because poor data produces poor models.
3
Alignment methods such as RLHF and DPO use preference data to move a model toward responses that people find more helpful, while DPO removes the need for a separate reward model.
Summary
Philipp Schmid explains how teams can take open LLMs from research into production by fine-tuning them for a specific task and aligning their behavior. He contrasts pre-training, which uses vast amounts of text and compute, with supervised fine-tuning, which uses instruction and response pairs. The quality of those examples matters: data should cover the required tasks, follow a consistent format, and receive human review. Schmid then explains why generative models are difficult to evaluate, since several answers can be correct while differing in usefulness. Alignment adds preference information, such as choosing between two Stack Overflow answers, so a model learns which style people prefer. He covers RLHF, AI-generated feedback, constitutional AI, and Direct Preference Optimization. DPO is easier to run because it trains directly on comparison data without a separate reward model. Hugging Face's TRL library provides SFT and DPO trainers, with support for parameter-efficient fine-tuning through methods such as LoRA and QLoRA.
Open LLMs have narrowed the gap with closed models
Schmid describes the rapid progress of open models from Llama and Alpaca through StarCoder, Falcon, Llama 2, Mistral 7B, and Mixtral. He points to the LMSYS Chatbot Arena, where users compare model responses, as evidence that open models can perform near capable closed models such as GPT-3.5 Turbo. This matters for production because teams can take an open model, run it themselves, and adapt it to their own data and use case. Schmid connects this progress to his belief that open source is needed to use generative AI responsibly.
Fine-tuning adapts a model with instruction and response pairs
Pre-training teaches a model about the world by predicting tokens across very large text collections. Fine-tuning keeps the next-token objective but changes the data into examples with an instruction and an expected output. The training process reduces the difference between the generated response and the provided answer. Schmid says fine-tuning needs much less compute and data than pre-training, and parameter-efficient methods reduce the memory requirement further. He gives the example that an LLM such as Llama or Mistral can be fine-tuned with a few thousand samples and a relatively small budget.
Generative models are harder to evaluate than classifiers
A classification model can often be scored with labels such as positive or negative. A generative model may produce a complete story or explanation, and several outputs can be correct while differing in usefulness. Schmid uses two answers to a Stack Overflow question about training XGBoost for multiple continuous outputs. Both answers are technically correct, but he prefers the one with documentation, a code snippet, and a more usable format. This makes evaluation dependent on the intended use and on the response qualities that people value.
Instruction data needs coverage, structure, and review
Schmid compares instruction data to traditional machine learning data with the phrase, "garbage in, garbage out." Instructions should state clearly what the model must do. The dataset should cover the tasks and topics the model will face, rather than containing mostly one kind of example. Formatting should remain consistent, such as the structured format used by Alpaca. Human feedback also helps distinguish merely acceptable responses from answers with the desired level of detail, structure, or behavior.
Synthetic instruction data can expand a domain dataset
Schmid describes Self-Instruct, where a strong model uses seed instructions to generate many instruction-following examples. Alpaca used this approach to create 50,000 examples from an initial instruction set. He also discusses Evol-Instruct, which makes seed prompts more complex. A simple prompt such as "one plus one" can become a question about when one plus one does not equal two. The same process can start from medical, agricultural, pharmaceutical, or other domain text, although the resulting data still needs inspection and filtering.
Hugging Face TRL simplifies supervised fine-tuning
Hugging Face's TRL library provides trainers for generative AI methods, including supervised fine-tuning. A user supplies a dataset with prompts and completions, while the trainer handles formatting, tokenization, and training settings. The SFT trainer supports parameter-efficient fine-tuning with LoRA and QLoRA through a configuration. Schmid recommends the library's end-to-end fine-tuning guide, which starts by checking whether a use case is suitable for fine-tuning and continues through training and evaluation.
Alignment uses preferences to shape model behavior
After supervised fine-tuning, alignment can move a model toward a preferred style or behavior. Schmid gives the example of comparing two technically correct answers and choosing the one that includes useful code and documentation. Preference data can contain pairwise choices or rankings among several outputs. RLHF uses this information to train the model toward responses preferred by people. The same idea can reduce harmful outputs or make responses more helpful, depending on the target behavior.
DPO provides a simpler alternative to classic RLHF
Classic RLHF can require a reward model, several models in the training setup, and more complex infrastructure. AI feedback can replace some human comparisons, with a model such as GPT-4 judging which of two outputs is better. Direct Preference Optimization takes a simpler route by training directly on pairwise comparison data, removing the separate reward-model step. Schmid says DPO is cheaper and is supported by a DPO trainer in TRL. He describes a Mistral example where the DPO model produced responses more likely to be preferred than the SFT model.
