# DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Adam Becker, MLOps Community & Nehil Jain, Stealth AI Startup & Matt Squire, Fuzzy Labs & Sophia Skowronski, Breckinridge Capital Advisors | MLOps Reading Group | 1:00:25

Source: https://www.youtube.com/watch?v=RQjJAr0EhKE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/deepseek-r1-incentivizing-reasoning-capability-in-llms-via-reinforcement
Published: 2025-03-05
Tags: distillation, reinforcement-learning, training-pipelines

## TL;DR
- The paper trains DeepSeek-R1 from DeepSeek-V3 with reinforcement learning, using rule-based rewards for tasks such as mathematics and coding.
- DeepSeek-R1 generates long chains of thought through reinforcement learning, then transfers that capability into smaller models through distillation.
- The paper leaves important parts of the training workflow difficult to reproduce, including the data preparation, infrastructure, experiment process, and collaboration around the models.

## Summary
The paper DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning describes how DeepSeek used reinforcement learning to produce a model that reasons through difficult problems with long chains of thought. Adam Becker first explains why next-token prediction and supervised fine-tuning struggle with unfamiliar puzzles, then covers test-time methods such as best-of-n, beam search, and lookahead search. Sophia Skowronski explains Group Relative Policy Optimization, rule-based rewards, KL divergence, and the use of format and accuracy checks. Nehil Jain walks through the transition from DeepSeek-R1-Zero to R1, including curated examples, supervised fine-tuning, language-consistency checks, and further reinforcement learning. Matt Squire discusses distillation, failed approaches such as process reward models and Monte Carlo tree search, and the paper's limited reproducibility from an MLOps perspective. The group also considers reasoning models in software agents and using a second model to format a reasoning model's output.

## Key ideas
### Scaling model size creates cost and data limits
[01:37](https://www.youtube.com/watch?v=RQjJAr0EhKE&t=97s)
Adam Becker describes the recent pattern of making language models larger and feeding them more data. He compares earlier models such as BERT and GPT-2, then points out that larger training runs cost more, take longer, and may eventually run out of usable data. He also mentions compute restrictions as a reason to seek more efficient ways to improve model capability. The problem is especially visible for reasoning tasks, where a model must try an approach, notice that it failed, retrace its steps, and try again.

### Supervised fine-tuning maps familiar inputs to familiar outputs
[04:41](https://www.youtube.com/watch?v=RQjJAr0EhKE&t=281s)
The presenters describe supervised fine-tuning as a paradigm that trains a model toward a single input-output mapping. That works for queries resembling the training distribution, such as answering that France's capital is Paris. It is less effective when a problem requires the model to move outside that distribution, experiment with possible solutions, and refine its answer. The discussion uses crossword-like and mathematical problems to illustrate why predicting the next token alone does not guarantee this kind of iterative reasoning.

### Test-time compute gives a model more ways to search for an answer
[05:38](https://www.youtube.com/watch?v=RQjJAr0EhKE&t=338s)
Adam Becker introduces work on scaling language-model computation at test time instead of only scaling model parameters. Best-of-n generates several attempts and selects one. Beam search breaks a solution into steps, continues promising branches, and drops others. Lookahead search simulates possible future steps and uses a reward to choose an approach. The group notes that these methods still depend on a verifier that can judge whether an intermediate step looks promising, and that the methods discussed were not yet very effective.

### DeepSeek-R1-Zero uses reinforcement learning with simple rewards
[12:33](https://www.youtube.com/watch?v=RQjJAr0EhKE&t=753s)
DeepSeek starts with DeepSeek-V3 and applies Group Relative Policy Optimization, or GRPO. Sophia Skowronski explains that GRPO removes the need for a separately trained value model, reducing memory and compute compared with PPO. The policy model generates several outputs, a reference model helps measure divergence, and reward functions score the outputs. For DeepSeek-R1-Zero, the rewards could use deterministic checks for mathematics and coding, along with format checks such as whether the output used the required think tags.

### The training process produced reasoning behavior without reasoning labels
[28:58](https://www.youtube.com/watch?v=RQjJAr0EhKE&t=1738s)
Skowronski says the R1-Zero process did not use training data that explicitly taught the model how to reason. Through the prompt and reinforcement-learning loop, the model began producing reasoning language and sometimes told itself to wait and reevaluate. The presenters connect this to a behavior that appeared during optimization rather than being directly supplied as labeled reasoning examples. They also report problems with language switching inside the reasoning output and poor readability.

### DeepSeek-R1 adds curated data and repeats the reinforcement-learning loop
[30:57](https://www.youtube.com/watch?v=RQjJAr0EhKE&t=1857s)
Nehil Jain explains that R1 addresses R1-Zero's readability and generalization problems through an iterative process. The team generated examples with other language models, filtered useful R1-Zero outputs, used human review for a small high-quality sample, and applied supervised fine-tuning. They then repeated reinforcement learning with accuracy, formatting, and language-consistency rewards. Additional data from DeepSeek-V3 broadened the process beyond mathematics and coding to English and more general reasoning tasks. The final model was also used to generate data for smaller distilled models.

### Distillation makes reasoning capability usable in smaller models
[46:09](https://www.youtube.com/watch?v=RQjJAr0EhKE&t=2769s)
Matt Squire explains that DeepSeek-R1 can act as a teacher for smaller models based on Qwen and Llama. The team generates training samples from R1 and fine-tunes the smaller models on those samples. The paper compares this with applying reinforcement learning directly to a smaller model. Squire reports the paper's conclusion that distilling a powerful reasoning model gives good results with less computation, while direct reinforcement learning on smaller models requires more compute and does not reach the same performance. The group sees this as useful for running reasoning models on less expensive infrastructure.

### The paper does not describe enough of the workflow to reproduce the results
[54:23](https://www.youtube.com/watch?v=RQjJAr0EhKE&t=3263s)
Squire's main MLOps concern is that the paper explains its methods and outcomes without showing enough of the surrounding workflow. The paper does not make clear how the authors prepared data, collaborated, managed experiments, or organized the infrastructure behind the iterative process. The group discusses the need for generic pipelines with interfaces for reward functions, models, search methods, and hardware choices. They also discuss the practical option of using one expensive model for reasoning and a cheaper model to extract structured output.

## Notable quotes
- Adam Becker: "The idea here is as they say similar to how humans think: if you ask me a very difficult question, I just need to give me a minute, I need to think about it, and then once I think about it I can come up with perhaps a better answer." (29:09)
- Sophia Skowronski: "No training data that explicitly told the model how to reason, just using this prompt and this reinforcement learning process, it was able to naturally reason or use anthropomorphic reasoning language in its think tags." (28:40)
- Nehil Jain: "Nothing like crazy new innovation in the R1 piece. I think this GRPO application in the zero was where they really figured out how to get a good quality policy." (36:22)
- Matt Squire: "For me, I still feel like that's something that's not discussed much in these papers around various large language models and how they've been trained and how they've been optimized." (54:44)

## Tools & references mentioned
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DeepSeek-V3
- DeepSeek-R1-Zero
- OpenAI o1
- OpenAI o1-mini
- BERT
- GPT-2
- Scaling Language Models at Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters
- PPO
- GRPO
- Qwen
- Llama
- DeepMind
- MLOps Community
- react-flow

## Who should watch
- You are trying to understand what DeepSeek-R1 changed in model training, especially the role of long chain-of-thought generation and reinforcement learning.
- You are comparing reinforcement learning with distillation and want the paper's practical trade-offs for smaller models.
- You work on LLM training pipelines and want to hear an honest discussion of what the paper does not reveal about reproducibility and experiment management.

## Editor's note

Matt Squire says the paper does not explain enough of the data preparation, experiment management, collaboration, and infrastructure behind DeepSeek-R1's iterative process. ZenML records each pipeline step, its inputs, outputs, and code version, so runs and artifacts can be traced to the data and code that produced them. Its configurable stack also lets the same workflow run on different infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [RAG: The Future of AI Search (and Prompt Engineering)](https://mlopstalks.com/talks/rag-the-future-of-ai-search-and-prompt-engineering) (Korri Jones, Chick-fil-A Corporate Support Center & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Sophia Skowronski, Breckinridge Capital Advisors & Lihu Chen & Binoy Perera, 10:13)
- [Building and Curating Datasets for RLHF and LLM Fine-tuning](https://mlopstalks.com/talks/building-and-curating-datasets-for-rlhf-and-llm-fine-tuning) (Daniel Vila Suero, Argilla, 58:51)
- [The Next Revolution in AI: LLMs and Beyond](https://mlopstalks.com/talks/the-next-revolution-in-ai-llms-and-beyond) (, 13:47)
- [LLMs Mini Summit // MLOps Mini Summit Meetup #3](https://mlopstalks.com/talks/llms-mini-summit-mlops-mini-summit-meetup-3) (Thomas Capelle, Weights & Biases & Boris Dayma, Craiyon & Jonathan Whitaker & Robbie McCorkell, Leap Labs, 1:45:28)
- [LLM Evaluation with Arize AI's Aparna Dhinakaran](https://mlopstalks.com/talks/llm-evaluation-with-arize-ais-aparna-dhinakaran) (Arize AI's Aparna Dhinakaran, 55:48)