Cameron Wolfe explains modern language models through three parts: transformers, pre-training, and alignment.
2
Pre-training teaches a decoder-only transformer to predict the next token from large amounts of raw text.
3
ChatGPT differs from earlier GPT models mainly through alignment methods such as supervised fine-tuning and RLHF, which train outputs toward human preferences.
Summary
Cameron Wolfe gives a short technical explanation of why ChatGPT works better than earlier language models. His framework has three parts. Transformers process a sequence by letting each token attend to earlier tokens, then applying a feed-forward transformation. Pre-training uses next-token prediction on large collections of raw internet text, and the same process generates text one token at a time. Alignment teaches the model to produce outputs that meet human criteria such as being helpful, harmless, and truthful. Wolfe describes supervised fine-tuning with examples written by people, followed by RLHF, where people rank generated responses. He connects these pieces to the progression from GPT and GPT-2 to GPT-3 and then ChatGPT. The explanation is intentionally high level, but it gives a usable account of what changed at each stage and why human preference data matters.
A three-part framework explains the jump from GPT-3 to ChatGPT
Wolfe organizes the explanation around transformers, pre-training, and alignment. The framework answers how language models improved from earlier GPT systems to ChatGPT while keeping the presentation understandable. He first explains the neural network architecture, then how models learn from raw text, and finally how they are tuned to produce answers people prefer. The progression gives each technical component a specific role instead of treating ChatGPT as a single mysterious system.
Modern generative language models mainly use the transformer's decoder
A transformer can have an encoder and a decoder. The encoder builds an understanding of input text, while the decoder produces text from that representation. Wolfe says modern generative LLMs use only the decoder because their job is to generate text. Within each decoder layer, masked self-attention lets each token look at the tokens before it. A feed-forward transformation then processes each token individually, giving the model information about both token relationships and individual tokens.
Pre-training teaches the model to predict the next token
The pre-training objective is next-token prediction. Given text such as "LLMs are cool," the model learns to predict what comes next. This is self-supervised because the target token already exists in the text, so people do not need to label every example. Developers can download large amounts of raw internet text and train large models on it. Wolfe connects the use of bigger models and bigger datasets with the improvement from GPT and GPT-2 to GPT-3.
Text generation repeats next-token prediction autoregressively
The model uses the same basic objective during generation. It predicts one token, adds that token to the input sequence, predicts the next token, and repeats the process. Wolfe describes this as an autoregressive loop. The model does not produce the entire answer in one step. It continually extends the sequence based on the tokens that are already present.
Alignment trains an LLM to produce text that matches a human user's preferences. Wolfe gives criteria such as helpful, harmless, and truthful. People can write examples from scratch, rank alternative outputs, or use both approaches. The aim is to teach the model to capture these criteria in its responses rather than only predict likely text from its pre-training corpus.
Supervised fine-tuning and RLHF solve different annotation problems
Supervised fine-tuning uses a dataset of good examples written or collected by people. Wolfe says this is difficult because producing highly aligned answers from scratch takes time. RLHF presents people with generated outputs and asks them to choose which is better. Ranking is easier for people than writing a complete answer, so modern LLM training often begins with a smaller supervised dataset, described by Wolfe as perhaps 20,000 to 30,000 examples, and then uses more comparison data for RLHF.
The model progression combines architecture, scale, and alignment
Wolfe maps each stage of the LLM progression to a training change. Transformer architecture led to systems such as GPT and GPT-2. Larger models trained with next-token prediction over more data produced GPT-3. ChatGPT added stronger alignment through supervised fine-tuning and RLHF, teaching the system to produce outputs that people judged desirable.
"All of LLM pre-training is based upon next token prediction which just trains a language model by looking at a bunch of raw text and training the model to predict the next word given previous words as input."05:23
Who should watch
You want a compact explanation of how ChatGPT differs from a general next-token language model.
You are an engineer who needs to explain transformers, pre-training, and alignment to a non-specialist audience.
You understand the basic idea of LLMs but want a simple connection between GPT-2, GPT-3, and ChatGPT.
