# Building and Curating Datasets for RLHF and LLM Fine-tuning

Daniel Vila Suero, Argilla | LLMs in Production 2023 | 58:51

Source: https://www.youtube.com/watch?v=Ezz_5csCJqI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-and-curating-datasets-for-rlhf-and-llm-fine-tuning
Published: 2023-07-10
Tags: evals, fine-tuning, human-in-the-loop, reinforcement-learning

## TL;DR
- Human feedback can cover prompts, model outputs, rankings, ratings, and other fields in an LLM data pipeline.
- Separating prompt collection from response writing can prevent unclear instructions from becoming low-quality training pairs.
- High-quality domain feedback and ongoing evaluation matter more as models become more capable and harder to assess automatically.

## Summary
Daniel Vila Suero explains how to collect and manage data for supervised fine-tuning and reinforcement learning from human feedback. He defines feedback broadly. It can rate prompts, write completions, compare responses, flag harmful content, or assess factuality. The talk follows the data lifecycle from prompt collection to supervised fine-tuning, preference modelling, reward models, and reinforcement learning. Daniel argues that prompt and response collection should often be separate because a labeler may fail to notice that an unclear prompt also produces a poor answer. He uses the Dolly dataset and Alpaca curation work to show how annotation mistakes can affect training data. He also describes Argilla's feedback interfaces and its use for monitoring LLM applications, collecting production interactions, and asking people to rate responses. The talk is honest about the limits of RLHF. It is unstable, does not solve world knowledge well, and needs additional safeguards in high-stakes settings.

## Key ideas
### Feedback can describe every part of an LLM data point
[03:21](https://www.youtube.com/watch?v=Ezz_5csCJqI&t=201s)
Daniel defines human feedback as information about an input, one or more outputs, or other parts of the pipeline. In Argilla, annotators can rate a prompt, rank competing responses, write natural-language comments, or answer binary and categorical questions such as whether content is harmful. This matters because prompt quality can affect training just as much as response quality. A dataset can be built through the Python SDK, with fields and annotation questions defined in advance. Multiple annotators can then work on the same records, and the collected feedback can be read back for training or evaluation.

### RLHF moves through distinct data and modelling stages
[16:53](https://www.youtube.com/watch?v=Ezz_5csCJqI&t=1013s)
The lifecycle Daniel presents starts with a base model trained on internet-scale data. Supervised fine-tuning uses demonstrations, where a prompt is paired with a written completion. Preference modelling then uses comparisons between responses, producing chosen and rejected examples for a reward model. In the final reinforcement learning stage, prompts are used to generate responses, and the reward model scores them. Human rankings are no longer used directly in that stage. Daniel also describes an iterative alternative, where new model generations are collected through a chat interface and reviewed repeatedly.

### Evaluation needs human review because automatic metrics miss important failures
[20:01](https://www.youtube.com/watch?v=Ezz_5csCJqI&t=1201s)
Daniel warns that a model can produce fluent, authoritative text that is inaccurate. He cites work showing that crowd workers may prefer imitation models because of their style, even when those models contain false information. He also argues that simply deploying a prompt-based application in shadow mode does not answer whether its performance is acceptable. Teams need ground-truth data, another evaluator, or feedback from users and experts. Argilla's callback can store interactions from a chain application in a dataset so people can rate the responses and calculate evaluation measures over time.

### Prompt collection should use real queries where possible
[29:24](https://www.youtube.com/watch?v=Ezz_5csCJqI&t=1764s)
Teams can gather prompts from existing resources, user queries, APIs, or experts who write new examples. Daniel recommends checking the quality and diversity of existing queries rather than assuming that a synthetic prompt set matches production. Asking people to write prompts about assigned topics can control topic distribution, but it can also create a gap between the dataset and the questions users really ask. Monitoring a live LLM application provides another source of real prompts, and good existing responses may become useful evaluation or fine-tuning examples.

### Separating prompt writing from response writing protects data quality
[33:39](https://www.youtube.com/watch?v=Ezz_5csCJqI&t=2019s)
Daniel questions the practice of asking one person to write both a prompt and its answer in the same step. A labeler may understand their own unclear instruction and produce a response that looks acceptable to them. If another person reviews the prompt separately, they can reject it when the instruction is confusing. He uses the Dolly dataset as an example of how misunderstanding a task can create bad records. In one information extraction example, a labeler copied a full Wikipedia article instead of providing the short context needed for the task. Daniel says the curation effort found more than 400 such issues, accounting for more than 10 percent of some task categories.

### Small, carefully curated instruction datasets can be useful
[41:11](https://www.youtube.com/watch?v=Ezz_5csCJqI&t=2471s)
Daniel contrasts several dataset sizes. The InstructGPT work used data from APIs and labelers, while Dolly collected examples from Databricks employees. He also discusses the paper 'LIMA: Less Is More for Alignment', which used 1,000 carefully curated instruction examples and reported strong results for following instructions and style. His conclusion is that teams may be able to start with about 1,000 or 2,000 high-quality examples rather than assuming they need tens of thousands. He does not claim that a small dataset solves every aspect of alignment.

### Preference data can support more than RLHF training
[44:19](https://www.youtube.com/watch?v=Ezz_5csCJqI&t=2659s)
For preference modelling, annotators compare several responses and rank them according to the task's definition of helpfulness, harmlessness, or another property. A ranking of several responses can be converted into pairs of chosen and rejected answers for training. Daniel says preference models can also help select among model snapshots when standard evaluation metrics show overfitting. They can evaluate responses after the fact as well. He mentions direct preference optimization as a newer approach that uses preference data without the same reinforcement learning process.

### RLHF has limits in high-stakes and knowledge-heavy settings
[53:18](https://www.youtube.com/watch?v=Ezz_5csCJqI&t=3198s)
In the questions, Daniel says RLHF is unlikely to solve world knowledge, sarcasm, or second-order logic by itself. A model may prefer one answer even when both answers are factually wrong, so external knowledge bases may be more promising for those problems. He also describes RLHF as unstable and recommends additional guardrails for high-stakes applications. At the time of the talk, Argilla's more dynamic feedback loops were still developing. He points to Anthropic's work on weekly updates as an example of a more iterative process, while saying that Argilla had not yet explored that approach fully.

## Notable quotes
- "We believe that feedback is not only about outputs but it can be also about inputs and it can be about many other things in the pipeline and in the data." (04:21)
- "If you separate those two kind of data collection processes, you can ask users to write prompts and then other users or maybe the same users to write the responses." (34:26)
- "We believe that human feedback and likely aided by machine feedback is key to deploy an aligned and robust LLM solution." (47:47)
- "I don't think that RLHF is the best way to solve this problem. I think using external knowledge bases and other things is much more promising." (54:32)

## Tools & references mentioned
- Argilla
- Hugging Face Spaces
- Python SDK
- Docker
- RLHF
- supervised fine-tuning
- instruction tuning
- Anthropic
- OpenAI
- TruthfulQA
- Alpaca
- Dolly
- Vicuna
- Falcon
- LIMA: Less Is More for Alignment
- InstructGPT
- Orca
- TRL
- Direct Preference Optimization
- LangChain
- Human in the Loop Machine Learning

## Who should watch
- You are collecting instruction or preference data and need a practical structure for prompts, completions, rankings, and annotation guidelines.
- Your LLM application is already receiving real user interactions, but you do not yet have a process for reviewing responses and turning that feedback into evaluation data.
- You are considering RLHF for a high-stakes system and need a clear account of its instability, data requirements, and limits.

## Related talks

- [RLHF Data Collection in Practice](https://mlopstalks.com/talks/rlhf-data-collection-in-practice) (Andrew Mauboussin, Surge AI, 12:10)
- [From Research to Production: Fine-Tuning & Aligning LLMs](https://mlopstalks.com/talks/from-research-to-production-fine-tuning-aligning-llms) (Philipp Schmid, Hugging Face, 38:03)
- [Evaluating and Integrating ML Models](https://mlopstalks.com/talks/evaluating-and-integrating-ml-models) (Morgan McGuire & Anish Shah, Weights & Biases, 51:57)
- [LLMs Mini Summit // MLOps Mini Summit Meetup #3](https://mlopstalks.com/talks/llms-mini-summit-mlops-mini-summit-meetup-3) (Thomas Capelle, Weights & Biases & Boris Dayma, Craiyon & Jonathan Whitaker & Robbie McCorkell, Leap Labs, 1:45:28)
- [All About Evaluating LLM Applications](https://mlopstalks.com/talks/all-about-evaluating-llm-applications) (Shahul Es, Exploding Gradients, 50:40)