AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression🔗9.1.1 DeepSeek-R1
DeepSeek-R1 mainly includes two inference models: DeepSeek-R1-Zero and DeepSeek-R1
- DeepSeek-R1-Zero is trained through large-scale reinforcement learning from the pre-trained model without requiring SFT before RL. During the RL process, DeepSeek-R1-Zero exhibited many strong and interesting inference behaviors.
- DeepSeek-R1 combines multi-stage training and cold-start data before RL to address issues with DeepSeek-R1-Zero (poor readability, language mixing, etc).
Performance: DeepSeek-R1 achieves performance on inference tasks comparable to OpenAI-o1-1217.
DeepSeek has open-sourced DeepSeek-R1-Zero, DeepSeek-R1, as well as six Dense Models distilled from DeepSeek-R1:
- 1.5B, 7B, 14B, 32B models based on Qwen
- 8B, 70B models based on LLaMA
Traditional RLHF + PPO training involves the following models (see Section 3.3.3 RLHF + PPO in LLM Alignment):
The training workflows in DeepSeek-R1 look like this:
A more detailed workflow:
The DeepSeek-R1-Distill is trained with the 800k data curated with previous stage of R1-Reasoning-RL and the DeepSeek-V3
<ins/>
Introduction
Research Background: Post-training has been shown to improve the accuracy of reasoning tasks, align better with social values, match user preferences, and require fewer computational resources than pre-training. OpenAI first proposed enhancing the CoT with longer reasoning processes and introduced inference-time scaling. Subsequent research explored effective inference-time scaling, including methods such as PRM (Process Reward Models), MCTS (Monte Carlo Tree Search), and Beam Search. However, none have reached the level of inference capability demonstrated by OpenAI-o1.
Research Method:
- DeepSeek-R1-Zero: This study explores pure Reinforcement Learning (RL) to enhance LLMs’ reasoning performance. Unlike previous approaches, it does not rely on SFT. Instead, it leverages the model’s self-evolution during the RL process.
- DeepSeek-R1: The DeepSeek-R1-Zero model, trained purely through RL, has issues such as poor readability and language mixing. This study addresses these by introducing a small amount of cold-start data and multi-stage training. Finally obtains DeepSeek-R1.
- Distillation: The study also explores the effect of distilling DeepSeek-R1 into smaller models.
Research Contributions:
- Large-scale RL on the base model
- By performing RL directly on Deepseek-V3-base, DeepSeek-R1-Zero demonstrates capabilities like self-verification, reflection, and long CoTs. It shows that LLMs can learn effectively through pure RL without SFT
- Introduce the pipeline to develop DeepSeek-R1
- Distillation shows small models can be powerful
- This study demonstrates that LLM reasoning can be successfully distilled into smaller models, which then outperform small models trained with RL alone.
- The open-sourced DeepSeek-R1 and its distilled checkpoints achieve state-of-the-art results.
DeepSeek-R1-Zero
RL plays an important role in reasoning tasks, but previous work heavily relied on large amounts of supervised data to improve model performance. Collecting such data is very resource-intensive.
Review of the GRPO Algorithm
DeepSeek-R1 uses GRPO for RL training.
To compute GAE, PPO requires three models: Reference, Reward, and Value. Instead, GRPO removes the Value model by using the average reward of multiple sampled outputs for the same question as an estimate of value. This simplifies the training process of RL.
GRPO’s loss function:
with
We can calculate the advantage by (ORM):
More details can be seen in 3.4.1 GRPO
Reward Modeling
Rewards are the source of training signals and determine the optimization direction of RL. This study designs a rule-based reward system that mainly consists of two types of rewards:
- Accuracy Rewards: Evaluate if the response is correct.
- Math problems with deterministic results: The model is required to provide the final answer in a specified format (e.g., within a box), which enables rule-based verification of correctness.
- Coding problem: Use a compiler to verify correctness based on predefined test cases.
- Format Rewards: This encourages the model to put its thinking process between
<think>and</think>tags.
⚠️ Due to the issue of reward hacking (LLMs learn behaviors that maximize the reward signal in deceptive ways), DeepSeek-R1-Zero does not use ORM or PRM reward models.
Training Template
This work designs a simple template to guide the basic model's reasoning process. The template requires DeepSeek-R1-Zero to first produce a reasoning process (within
<think> and </think> tags), followed by the final answer. This study only sets up the structure of the task, without forcing the model to think in a specific way or follow certain strategies. That way, we can see how the model naturally learns and improves during training.
Performance of DeepSeek-R1-Zero
Reinforcement learning enables DeepSeek-R1-Zero to achieve strong reasoning capabilities without any SFT data.
- As RL training progresses, the results on the AIME benchmark show that DeepSeek-R1-Zero’s performance increases consistently.
- The performance of DeepSeek-R1-Zero can be further enhanced through majority voting (see
cons@64in the result below).
cons@64indicates consensus at 64 independent tries
Self-evolution
- As the number of training steps increases, the model’s response length or thinking time tends to grow linearly, with a noticeable presence throughout the entire training process. This is not an explicitly programmed outcome, but rather the result of self-evolution within the model.
- One of the most notable aspects of this self-evolution is that, with more inference time, complex behaviors begin to emerge. For example, reflective behaviors, where the model revisits and reevaluates prior steps, or exploratory behaviors, where it tries alternative methods to solve problems. These behaviors are not hardcoded but emerge from the model’s interaction with a reinforcement learning environment.
Aha Moment of DeepSeek-R1-Zero
- This "aha moment" is when the model learns to re-evaluate initial approaches and allocate more time to think about the problem.
- The example below illustrates the power and beauty of reinforcement learning: rather than explicitly teaching the model how to solve a problem, this study simply sets up rewards that guide the model, and it figures out problem-solving strategies on its own.
<ins/>
DeepSeek-R1
Based on the DeepSeek-R1-Zero model, this study raises two research questions:
- By adding a small amount of high-quality data as a cold start, can we further improve reasoning ability or accelerate convergence?
- How do we train a user-friendly model that not only produces clear and coherent CoT, but also demonstrates strong generalization ability?
Stage 1: Cold Start
To prevent instability during early RL training from the pre-trained base model, for DeepSeek-R1, this study constructed and collected a small amount of long CoT data to fine-tune the base model. Then use this SFT model for the later RL training. The methods for collecting CoT include:
- Using a few-shot prompting with a long CoT as an example
- Directly prompting models to generate detailed answers with reflection and verification
- Gathering DeepSeek-R1-Zero outputs in a readable format and refining the results through post-processing by human annotators.
This work collected thousands of cold-start data points.
This method is common in RL research, especially in hard-to-train tasks or algorithms with unstable early training phases. It’s typical to pre-train with a small amount of high-quality demonstration data to obtain a decent initial policy, and then continue with reinforcement learning.
Advantages of Cold-Start Data: Responses generated by DeepSeek-R1-Zero are often hard to read (language mixing or lack of a Markdown format for user-friendly display). To address this, the authors designed a readable format when creating cold-start data, i.e., each answer ends with a summary:
|special_token|<reasoning_process>|special_token|<summary>, and filters out user-unfriendly content.Stage 2: Reinforcement Learning for Reasoning
After supervised fine-tuning the DeepSeek-V3-Base with cold-start data, this part applies the same RL (i.e. GRPO) process used in DeepSeek-R1-Zero. This stage focuses on enhancing the model’s reasoning capabilities.
- To solve mixed-language issues in CoT output, this study introduces a language consistency reward, which is simply added to the reasoning accuracy reward.
- The reward is computed based on the proportion of target language words in the CoT.
Stage 3: Rejection Sampling and SFT
Sage 1 and Stage 2 compose a complete SFT + RL process, and the result shows improved readability. This stage aims to start a new round of the SFT + RL process with more high-quality and diverse data for SFT training.
So, instead of using only reasoning data, it starts to collect data from other domains to enhance the model’s capabilities in writing, role-playing, and general-purpose tasks.
- Reasoning Data: The study collects reasoning prompts and uses rejection sampling to select generated responses. It includes two types of data:
- Data with a clear solution (for example, a single answer for a math problem): Use a rule-based reward to evaluate the response.
- Data without a clear solution: Use DeepSeek-V3 to evaluate the response. Feeding DeepSeek-V3 with ground truth and the model’s response, ask DeepSeek-V3 to tell whether the response is correct or not.
For each prompt, multiple responses are sampled, and only correct responses are kept.
Total: 600K reasoning data samples
- Non-reasoning Data: For data like writing, factual QA, self-cognition, and translation, this study adopts the DeepSeek-V3 pipeline and reuses portions of the SFT dataset of DeepSeek-V3:
- Simple queries (like
hello): Do not provide a CoT in response. - Other complex queries: Prompting DeepSeek-V3 to provide CoT before answering the question.
Total: 200K data samples
Finally, DeepSeek-V3-Base was trained on these 800K samples for two epochs of SFT.
Stage 4: Reinforcement Learning for all Scenarios
The second-stage reinforcement learning aims to improve the model’s Helpfulness and Harmlessness while further enhancing its reasoning ability.
Reward design:
- For reasoning data (math, code, and logical reasoning): rule-based rewards.
- For general-purpose data (complex and nuanced scenarios): use a Reward model. This model is built upon the preference data from the DeepSeek-V3 pipeline and adopts a similar distribution of preference pairs and training prompts.
Remember that the entire response is composed of a reasoning process (the content between the
<think> tab) and the final summary:- For Helpfulness: Focus only on the final summary, ensuring that the evaluation emphasizes the utility and relevance of the response to the user, while minimizing interference with the underlying reasoning process.
- For Harmlessness: Evaluate the entire response to detect and reduce any potential risks, biases, or harmful content during generation.
Distillation: Empower Small Models with Reasoning Ability
This paper performs SFT using 800K samples in Stage 3 on Qwen and LLaMA models directly. The results show that the knowledge distillation method significantly improves the reasoning capabilities of small models. The base models used for distillation include:
- Qwen2.5-Math-1.5B
- Qwen2.5-Math-7B
- Qwen2.5-14B
- Qwen2.5-32B
- LLaMA-3.1-8B
- LLaMA-3.3-70B-Instruct
<ins/>
Experiment
DeepSeek-R1 Performance
- In Chinese SimpleQA, DeepSeek-R1 performed worse than DeepSeek-V3, mainly because after safety RL, it tends to refuse to answer certain queries. Without safety RL, DeepSeek-R1 can reach over 70% accuracy.
- The summary lengths generated by DeepSeek-R1 are concise: Average of 689 tokens on ArenaHard, and 2218 tokens on AlpacaEval 2.0. This indicates that DeepSeek-R1 avoids introducing length bias (a preference for longer responses) during evaluation.
- DeepSeek-R1’s performance is expected to improve in the next version because the current amount of RL training data is still quite limited.
Distillation Experiment Results
- Simply distilling DeepSeek-R1’s outputs enables the efficient DeepSeek-R1-7B (i.e., DeepSeek-R1-Distill-Qwen-7B) to outperform non-reasoning models like GPT-4o-0513 across the board.
- DeepSeek-R1-14B surpasses QwQ-32B-Preview on all evaluation metrics, while DeepSeek-R1-32B and DeepSeek-R1-70B outperform o1-mini on most benchmarks.
- Applying RL to these distilled models yields further improvements.
Discussion
Distillation vs. Reinforcement Learning
The Qwen-32B-Base model, after large-scale RL training, achieves similar performance to QwQ-32B-Preview. However, DeepSeek-R1-Distill-Qwen-32B outperformed DeepSeek-R1-Zero-Qwen-32B across all benchmarks. Conclusion:
- Distillating smaller models with powerful models can produce better performance.
- Smaller models trained by RL require more computational power than distilling and may fail to match the performance of distilled models.
- Although distillation is economical and efficient, pushing the boundaries of intelligence may still require stronger base models and larger-scale reinforcement learning.
Unsuccessful Attempts
- Process Reward Model (PRM):
- For general reasoning tasks, clearly defining fine-grained steps is challenging.
- Determining if the current intermediate step is correct is challenging. Using a model for auto-labeling may yield unsatisfactory results, while manual labeling is hard to scale.
- Once PRM is introduced, it inevitably leads to reward hacking, and retraining reward models demands significant additional training resources, complicating the training pipeline.
- Monte Carlo Tree Search (MCTS):
- Unlike AlphaGo/AlphaZero used for Go/chess (search space is relatively well-defined), tokens generated by LLMs have a vast, exponential search space. To address this, a maximum extension limit is set for each node, but this can lead to the model getting stuck in local optimal solutions.
- The Value model evaluates the quality of each generated token at every search step, but the training of high-granularity value models is challenging, making it hard to improve the model.
- While AlphaGo’s success relied on training a Value model to progressively enhance its performance, this principle proves difficult to replicate in LLM due to the complexities of token generation.
Future Directions
- Generalization Ability: DeepSeek-R1 shows a gap compared to DeepSeek-V3 in tasks such as function calling, multi-turn dialogue, complex role-playing, and JSON output. Future work should explore how to leverage long CoT to enhance performance in these areas.
- Language Mixing: R1 sometimes uses English to generate reasoning and responses regardless of whether the question is in Chinese or English.
- Prompt Engineering: During the evaluation of DeepSeek-R1, the model was observed to be highly sensitive to prompts. Few-shot prompts consistently reduced performance. It is recommended to let users directly describe their questions and use zero-shot with a defined output format for best results.
- Software Engineering Tasks: Due to long evaluation times, DeepSeek-R1 shows limited gains over DeepSeek-V3 on software engineering benchmarks. Future versions aim to improve efficiency and performance through rejection sampling and asynchronous evaluation during RL.
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...