Supervised fine-tuning establishes the assistant behaviors that the model should learn, including when to answer, refuse, or say that it does not know.
2
Ranking model outputs is difficult because factuality, writing quality, length, and helpfulness can conflict, so labeler instructions need to state what matters most.
3
Surge AI lets labelers edit bad completions before ranking them, which creates a comparison involving a good response instead of rewarding one of several bad answers.
Summary
Andrew Mauboussin explains how Surge AI collects data for the two main stages of RLHF. Supervised fine-tuning uses thousands to tens of thousands of prompt-completion pairs to establish the assistant's intended behaviors. If the examples never include uncertainty or refusal, the model may confidently invent an answer, such as a restaurant address it does not know. Reward modeling then has labelers rank several model outputs, but rankings become difficult when one response is well written yet factually wrong and another is accurate but poorly written. Surge's labeling software lets a labeler edit an answer when every sampled completion is bad, then compare the edited response against another option. Mauboussin also describes research where labelers mark specific factual and relevance errors in text, producing more detailed signals that can be weighted later. The talk is short, practical, and honest about how much the quality of RLHF depends on the data collection interface and labeling instructions.
Surge AI treats RLHF data collection as a full-stack product
Andrew Mauboussin describes Surge AI as a human feedback company that hires its own contractors, builds the labeling platform they use, and delivers the resulting data to clients. The company works with groups including Anthropic, OpenAI, and Google. Its RLHF work covers the data collection process rather than only the model training step. Mauboussin frames the talk around two parts of that process: supervised fine-tuning and ranking model outputs.
A base language model continues text instead of acting like an assistant
A model trained only to predict the next word on internet text may not follow a request to write a story. When prompted for a Harry Potter story involving Voldemort, it might continue writing instructions about the story rather than produce the story itself. RLHF changes this behavior so the model responds as an assistant. Mauboussin uses GPT-4 as an example of a model that generates the requested content after similar training.
Supervised fine-tuning establishes the assistant's desired behaviors
The first stage uses a few thousand to tens of thousands of prompt-completion pairs. Labelers write the desired response from scratch, and the model is fine-tuned on those examples. The data should include every behavior the assistant needs, such as answering directly, using a more verbose style, or adding a disclaimer for medical or financial advice. Mauboussin calls this stage the foundation for the assistant's possible outputs.
Models can invent answers when training data never shows uncertainty
Mauboussin shows a Llama model that received instruction tuning but never saw an example where it could say, 'I don't know.' When asked for the address of a specific Italian restaurant in San Francisco, the model invents a random address because every training example expected it to answer the question. The example shows why uncertainty and refusal behaviors need to appear explicitly in the fine-tuning data.
Ranking outputs requires clear priorities from the labeler
In reward modeling, labelers compare sampled outputs and indicate which one is better. That decision can involve several competing factors. One answer may be polished, well structured, and the right length while containing a few factual errors. Another may contain the correct facts but have sloppy writing. Mauboussin says instructions must clearly explain which qualities to prioritize and what behavior the team wants.
Editing lets labelers avoid rewarding two bad answers
Sometimes every sampled completion is wrong. Mauboussin gives an example involving the most recent John Wick movie, where both responses incorrectly refer to John Wick 3 even though John Wick 4 had been released. Surge's labeling software includes an edit button. A labeler can rewrite one response to acknowledge the limits of the model's training data, then rank that good edited response against a bad source response.
Marking specific errors could produce more flexible reward signals
Mauboussin points to work from the Allen Institute where labelers mark factual errors and irrelevant text directly in a response. These annotations provide more detail than choosing one complete answer over another. A reward model could combine the marked problems in different ways, allowing a team to decide later how strongly to penalize factual mistakes relative to other qualities.
"It can be really hard to manage that, and this is what we found is really important, to be very clear in your instructions to people about what you're prioritizing and what your desired behavior is."08:08
Who should watch
You are building an RLHF pipeline and need to decide what labelers should collect during supervised fine-tuning.
Your ranking task often contains several bad model outputs and you need a way to avoid rewarding the least bad answer.
You want practical examples of how factuality, writing quality, uncertainty, and editing affect human feedback data.
