DeepSeek-R1: Reasoning Emerges from Reinforcement Learning Alone
DeepSeek-R1 proves that chain-of-thought reasoning can emerge from pure reinforcement learning without any supervised fine-tuning as a prerequisite, then packages that result into an open-weight, MIT-licensed model family anyone can run or distill.
For years, the accepted recipe for teaching a language model to reason treated supervised chain-of-thought data as load-bearing — that RL needed an existing behavioral scaffold to optimize around. DeepSeek-R1-Zero removes that scaffold entirely and produces a competitive reasoning model anyway.
Source
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
By: DeepSeek-AI
Published: Jan 20, 2025
Type: Technical report
Two things make this worth saving: the R1-Zero experiment challenges the accepted account of what supervised fine-tuning is actually for, and the open MIT license means the entire model family — including six distilled variants from 1.5B to 70B parameters — is freely usable, commercially deployable, and distillable without restriction.
How Reinforcement Learning Teaches a Model to Think
The orthodox recipe for making a language model reason looks like this: start with a strong base model, collect thousands of high-quality chain-of-thought demonstrations from annotators or a stronger model, run supervised fine-tuning on those traces, and then apply reinforcement learning to refine the outputs. The supervised step is treated as load-bearing. It provides the behavioral scaffold — a starting template of what reasoning looks like — that RL then polishes.
DeepSeek-R1-Zero removes that scaffold entirely. The team applied GRPO (Group Relative Policy Optimization) directly to DeepSeek-V3-Base, a 671B mixture-of-experts model that activates 37B parameters per forward pass, with a reward signal that was blunt and simple: get the math, code, or reasoning answer correct. No annotated reasoning traces. No curated demonstrations of step-by-step thinking. No cold-start dataset of any kind. Just the reward.
What happened next is the interesting part. The model, optimizing for correct answers, spontaneously started generating long chains of thought. It began checking its own work — producing a candidate answer, questioning it, and revising. It learned to allocate more computation to harder problems by generating more reasoning tokens before committing to an answer. These behaviors — self-verification, reflection, adaptive compute allocation — are precisely the behaviors that expensive supervised pipelines are designed to install. In R1-Zero, they appeared from the reward signal alone.
That observation warrants sitting with carefully. Chain-of-thought is not an arbitrary behavior that humans had to demonstrate and hand-transfer into a model. It is something the model discovers, through trial and error, because thinking out loud is instrumentally useful for getting correct answers. The structure of good reasoning is learnable from outcomes alone. The supervised data that traditional post-training pipelines treat as essential was, in a meaningful sense, doing what RL would have done anyway — just faster, and with significantly more annotation labor involved.
The catch is that purely RL-trained models hit real deployment failure modes. R1-Zero sometimes switches languages mid-reasoning, lapsing between English and Chinese unpredictably within a single chain of thought. It occasionally enters repetition loops, generating near-redundant reasoning text without terminating cleanly. Its output formatting is inconsistent enough to make downstream use awkward. These are not capability failures — the model is actually solving the underlying problems — but they are deployment failures that would make building anything on top of R1-Zero uncomfortable.
DeepSeek-R1 is the corrected version. The team kept the RL training but added two targeted interventions. A small cold-start dataset of a few thousand high-quality reasoning examples — fed to the model before RL begins — establishes format consistency without preinstalling reasoning capability. A multi-stage pipeline then runs two RL stages, the second targeting human preference alignment, and two SFT stages, the second for non-reasoning general capability. The cold-start data does not teach the model to reason; R1-Zero already knows how. It teaches the model to present reasoning in a consistent, readable form. This is a narrower and more honest claim about what supervised data is buying in post-training pipelines.
On benchmarks, DeepSeek-R1 matches or exceeds OpenAI o1-1217 in several areas: AIME 2024 at 79.8% compared to o1’s 79.2%, MATH-500 at 97.3% against o1’s 96.4%, and a Codeforces rating of 2029 against o1’s 2061. It pulls ahead on MMLU-Redux at 92.9% and AlpacaEval 2.0 at 87.6% LC win-rate. The overall picture is a frontier-tier reasoning model released under the MIT license with no restrictions on commercial use, fine-tuning, or distillation into other models.
Distillation is where the release has the most immediate practical reach. The team used R1’s generated reasoning traces to fine-tune smaller open models: checkpoints based on Qwen2.5 and Llama3 at six sizes from 1.5B to 70B parameters. The 32B distilled variant outperforms o1-mini across multiple benchmarks despite running at a fraction of its inference cost. A smaller model does not need to undergo RL training to acquire R1-level reasoning behaviors — it just needs supervised fine-tuning on R1’s outputs. Reasoning capability unlocked through large-scale RL at 671B scale compresses into smaller models through the right training data. That transfer is essentially free once R1’s checkpoints are public.
The practical implication is direct: frontier-quality reasoning capability is now available at 7B or 14B parameter scale, downloadable from Hugging Face, runnable on consumer hardware, at no API cost. One operational detail worth knowing: the R1 series works best without a system prompt — all instructions should go in the user turn, not a system message. For math problems, explicitly asking for step-by-step reasoning and a boxed final answer improves output stability. Temperature between 0.5 and 0.7 prevents the repetition loops that pure RL training can surface. These are not arbitrary constraints; they reflect how the model’s training shaped its generation dynamics.
My take
The R1-Zero result rewrites the conventional account of why supervised chain-of-thought data is collected. Models do not need human reasoning demonstrations to learn to reason — they can discover it from a reward signal. What supervised fine-tuning actually provides in the traditional pipeline is format stability and failure-mode mitigation, not the underlying reasoning capability. That reframing has real consequences for how post-training pipelines get designed and what annotation labor is actually worth paying for going forward.
Where this is useful
- Deploying a reasoning-capable model locally at 7B–32B scale to avoid cloud API costs for math, code, or logic tasks
- Using distilled R1 checkpoints as a drop-in reasoning upgrade in existing Qwen2.5 or Llama3 fine-tuning pipelines
- When NOT applicable: tasks that depend heavily on complex system prompt instructions — R1 was trained without them and performs better without them
Apply this: Before routing a math or reasoning task to a frontier API, benchmark a distilled R1 variant locally — the 7B or 14B checkpoints often close the performance gap at a fraction of the latency and cost.
What I want to remember
Remember: Chain-of-thought is emergent from reward, not instructed — supervised data’s real job in post-training is formatting and failure-mode mitigation, not the reasoning capability itself.