The paper trains DeepSeek-R1 from DeepSeek-V3 with reinforcement learning, using rule-based rewards for tasks such as mathematics and coding.
2
DeepSeek-R1 generates long chains of thought through reinforcement learning, then transfers that capability into smaller models through distillation.
3
The paper leaves important parts of the training workflow difficult to reproduce, including the data preparation, infrastructure, experiment process, and collaboration around the models.
Summary
The paper DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning describes how DeepSeek used reinforcement learning to produce a model that reasons through difficult problems with long chains of thought. Adam Becker first explains why next-token prediction and supervised fine-tuning struggle with unfamiliar puzzles, then covers test-time methods such as best-of-n, beam search, and lookahead search. Sophia Skowronski explains Group Relative Policy Optimization, rule-based rewards, KL divergence, and the use of format and accuracy checks. Nehil Jain walks through the transition from DeepSeek-R1-Zero to R1, including curated examples, supervised fine-tuning, language-consistency checks, and further reinforcement learning. Matt Squire discusses distillation, failed approaches such as process reward models and Monte Carlo tree search, and the paper's limited reproducibility from an MLOps perspective. The group also considers reasoning models in software agents and using a second model to format a reasoning model's output.
Adam Becker describes the recent pattern of making language models larger and feeding them more data. He compares earlier models such as BERT and GPT-2, then points out that larger training runs cost more, take longer, and may eventually run out of usable data. He also mentions compute restrictions as a reason to seek more efficient ways to improve model capability. The problem is especially visible for reasoning tasks, where a model must try an approach, notice that it failed, retrace its steps, and try again.
Supervised fine-tuning maps familiar inputs to familiar outputs
The presenters describe supervised fine-tuning as a paradigm that trains a model toward a single input-output mapping. That works for queries resembling the training distribution, such as answering that France's capital is Paris. It is less effective when a problem requires the model to move outside that distribution, experiment with possible solutions, and refine its answer. The discussion uses crossword-like and mathematical problems to illustrate why predicting the next token alone does not guarantee this kind of iterative reasoning.
Test-time compute gives a model more ways to search for an answer
Adam Becker introduces work on scaling language-model computation at test time instead of only scaling model parameters. Best-of-n generates several attempts and selects one. Beam search breaks a solution into steps, continues promising branches, and drops others. Lookahead search simulates possible future steps and uses a reward to choose an approach. The group notes that these methods still depend on a verifier that can judge whether an intermediate step looks promising, and that the methods discussed were not yet very effective.
DeepSeek-R1-Zero uses reinforcement learning with simple rewards
DeepSeek starts with DeepSeek-V3 and applies Group Relative Policy Optimization, or GRPO. Sophia Skowronski explains that GRPO removes the need for a separately trained value model, reducing memory and compute compared with PPO. The policy model generates several outputs, a reference model helps measure divergence, and reward functions score the outputs. For DeepSeek-R1-Zero, the rewards could use deterministic checks for mathematics and coding, along with format checks such as whether the output used the required think tags.
The training process produced reasoning behavior without reasoning labels
Skowronski says the R1-Zero process did not use training data that explicitly taught the model how to reason. Through the prompt and reinforcement-learning loop, the model began producing reasoning language and sometimes told itself to wait and reevaluate. The presenters connect this to a behavior that appeared during optimization rather than being directly supplied as labeled reasoning examples. They also report problems with language switching inside the reasoning output and poor readability.
DeepSeek-R1 adds curated data and repeats the reinforcement-learning loop
Nehil Jain explains that R1 addresses R1-Zero's readability and generalization problems through an iterative process. The team generated examples with other language models, filtered useful R1-Zero outputs, used human review for a small high-quality sample, and applied supervised fine-tuning. They then repeated reinforcement learning with accuracy, formatting, and language-consistency rewards. Additional data from DeepSeek-V3 broadened the process beyond mathematics and coding to English and more general reasoning tasks. The final model was also used to generate data for smaller distilled models.
Distillation makes reasoning capability usable in smaller models
Matt Squire explains that DeepSeek-R1 can act as a teacher for smaller models based on Qwen and Llama. The team generates training samples from R1 and fine-tunes the smaller models on those samples. The paper compares this with applying reinforcement learning directly to a smaller model. Squire reports the paper's conclusion that distilling a powerful reasoning model gives good results with less computation, while direct reinforcement learning on smaller models requires more compute and does not reach the same performance. The group sees this as useful for running reasoning models on less expensive infrastructure.
The paper does not describe enough of the workflow to reproduce the results
Squire's main MLOps concern is that the paper explains its methods and outcomes without showing enough of the surrounding workflow. The paper does not make clear how the authors prepared data, collaborated, managed experiments, or organized the infrastructure behind the iterative process. The group discusses the need for generic pipelines with interfaces for reward functions, models, search methods, and hardware choices. They also discuss the practical option of using one expensive model for reasoning and a cheaper model to extract structured output.
"The idea here is as they say similar to how humans think: if you ask me a very difficult question, I just need to give me a minute, I need to think about it, and then once I think about it I can come up with perhaps a better answer."Adam Becker29:09
Who should watch
You are trying to understand what DeepSeek-R1 changed in model training, especially the role of long chain-of-thought generation and reinforcement learning.
You are comparing reinforcement learning with distillation and want the paper's practical trade-offs for smaller models.
You work on LLM training pipelines and want to hear an honest discussion of what the paper does not reveal about reproducibility and experiment management.
