AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression🔗9.1.8 QwenLong-L1
In recent years, Large Reasoning Models (LRMs) have demonstrated strong reasoning capabilities through reinforcement learning (RL). These improvements mainly exist in short-context reasoning tasks. In contrast, extending LRMs to effectively handle and reason over long-context inputs via RL remains an unsolved challenge. QwenLong-L1 is the first to formalize long-context reasoning RL and identify key challenges such as suboptimal training efficiency and unstable optimization.
Training pipeline: warm-up SFT → Curriculum-guided phased RL (using a difficulty-aware reward sampling strategy for efficient exploration)
Experimental results: On seven long-context QA benchmarks, QwenLong-L1-32B outperforms closed source LRMs OpenAI-o3-mini and open source model Qwen3-235B-A22B, with performance comparable to Claude-3.7-Sonnet-Thinking.
Introduction
Overview
OpenAI-o1, DeepSeek-R1, and Qwen-QwQ are representative large reasoning models (LRMs), which enhance various reasoning tasks through RL exploration and training.
After RL training, LRMs exhibit signs of "slow thinking", characterized by the emergence of sophisticated problem-solving strategies like divide-and-conquer and loopback mechanisms in their extended chain-of-thought (CoT) reasoning output.
This process has improved models' performance on short-context tasks (4K tokens), but expansion to long-context scenarios (120K tokens) that require robust contextual grounding and multi-step reasoning remains not explored. This limitation poses a significant barrier to real-world applications involving interaction with external knowledge, such as deep research, where LRM must gather and process information in complex, knowledge-dense environments.
This paper proposes a concept for long-context reasoning RL: LRMs retrieve and ground relevant information from long-context inputs, then generate reasoning chains based on that information.
However, experimental results show that compared to short-context reasoning, long-context RL suffers from lower training efficiency, as illustrated in the figure below, because:
- Delayed reward convergence (figure (a), the blue curve). It originates from a significant drop in output entropy (figure (b)) when processing long‐context inputs, which limits policy exploration.
- Unstable optimization process, reflected by intermittent spikes in KL divergence (figure (c)). The instability arises because the wide distribution of input lengths leads to a much longer output length, amplifying variance effects and causing greater fluctuations during policy updates.
QwenLong-L1 Overview
A progressive context expansion strategy mainly consists of three components:
- Warm-up SFT to initialize a robust policy.
- Curriculum-guided RL phase to adapt from short to long context.
- Difficulty-aware retrospective sampling mechanism that changes the difficulty over time to encourage the model to explore different actions.
Meanwhile, the QwenLong-L1 framework integrates a hybrid reward function based on rules and models, balancing precision and recall.
Long-Context Reasoning RL
Recall - KL divergence
This paper extends the KL-constrained reinforcement learning objective by incorporating long-context information into the formula:
Where:
- is a short question (input),
- is the model’s output,
- is the long context.
Note that compared with standard RL (see Section 3.3.3 RLHF + PPO in LLM Alignment - Reference Model), the only difference is to add context .
The template for long-context reasoning problems is detailed below:
Recall - RL Algorithms
This section briefly recalls Proxy Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO). You can review them from:
- GRPO: Section 3.4.1 GRPO
- DAPO: Section 3.4.5 DAPO
For long-context inputs, the quadratic complexity of attention mechanisms makes training value networks computationally expensive. Therefore, this study primarily adopts the group-relative RL algorithm, including both GRPO and DAPO algorithms to estimate the advantage function and remove the need for a value model.
Progressive Context Scaling
To address the optimization instability mentioned earlier, the training framework of QwenLong-L1 is divided into three stages:
- Warm-Up SFT: Uses a high-quality demonstration dataset that distilled from a teacher LRM, where each sample includes a question , context , and a gold-standard reasoning path with verified correctness. SFT is used for initialization to obtain stable starting parameters, enabling basic context understanding, reasoning chain generation, and answer extraction. The SFT objective is defined as:
- Curriculum-Guided Phased Reinforcement Learning: The RL process is divided into discrete stages. Each stage targets a specific maximum context length . Starting from the initial input length , each stage gradually increases the input length until the maximum target length is reached. During phase , training is only conducted on samples satisfying the conditions:
- Difficulty-Aware Retrospective Sampling: This involves importance sampling weighted based on difficulty scores to selective retrospective samples (replaying past training examples). Specifically, the difficulty score is defined as the inverse of the average group reward for that sample:
Lower average rewards correspond to higher difficulty scores, and such samples are prioritized during replay.
Hybrid Reward Mechanism
In open-domain QA and long-context reasoning tasks, answers often exhibit diversity. If the rule-based rewards are too strict, they can block perfectly good answers and hurt the results. This work combines two types of rewards:
- Rule-Based Validation: ensures precision by checking whether the model’s output strictly adheres to the task-specific correctness criteria. For QA tasks, the final answer is extracted (using regular expression) from the generated output , and exact string matching is done with the gold reference answer :
Note: This work omits formatting rewards for answer extraction. Excessively rigid formatting rewards may oversimplify the learning objective, limiting reasoning chain exploration.
- LLM-as-a-Judge: To address false negatives caused by strict string matching, an LLM is used to assess the semantic equivalence between the generated answer and the reference answer, yielding a model-based reward:
The prompt template for evaluation is shown below.
The final reward is the maximum of the two rewards:
Training
Training Datasets
- RL Dataset: This study develops DocQA-RL-1.6K, an RL dataset with a total of 1.6K samples, covering three reasoning domains:
- Mathematical Reasoning: 600 questions from DocMath
- Logical Reasoning: 600 multiple-choice questions generated by DeepSeek-R1, focusing on reasoning over real documents in fields such as legal, financial, insurance, and production
- Multi-Hop Reasoning: 400 samples (200 each) from MultiHopRAG and Musique, emphasizing cross-document reasoning
- SFT Dataset: 5300 high-quality (question, document, answer) triplets distilled from DeepSeek-R1. Samples were filtered based on question quality, complexity, and diversity. Document quality and length were also controlled to ensure precise context information.
Training Details
- Base model: R1-Distill-Qwen-14B and R1-Distill-Qwen-32B
- SFT Training: Input length has 20K tokens
- RL Training: Two-phase curriculum-based context scaling:
- Phase 1 input length: 20K tokens
- Phase 2 input length: 60K tokens
This study uses a “difficulty-aware” sampling step that selects the most difficult examples (those with an average accuracy of zero from Phase I) and includes them again in Phase II.
Experiment
Key Experimental Results
The main experimental results on long-context DocQA benchmarks are summarized below, along with several key findings:
- Limited Efficacy of SFT in Long-Context Reasoning: The SFT model improved by only 0.8 percent on average with 14B parameters, and by 3.2 percent with 32B parameters. Furthermore, the improvements varied significantly across different benchmarks.
- Significant Improvements Via RL Integration:
- The 14B model has an average improvement of 4.0–4.1 percentage points.
- The 32B model has an average improvement of 4.7–5.1 percentage points.
- Leading Performance Among Mainstream LRMs:
QWENLONG-L1-14B scored an average of 68.3, outperforming Gemini-2.0-Flash-Thinking, R1-Distill-Qwen-32B, and Qwen3-32B, and performing on par with QwQ-32B.
QWENLONG-L1-32B achieved an average score of 70.7, outperforming QwQ-Plus, Qwen3-Plus, Qwen3-235B-A22B, and OpenAI-o3-mini, and even matching Claude-3.7-Sonnet-Thinking.
- Additional Enhancements by Test-Time Scaling: As the sampling scale increases, the performance of QWENLONG-L1-14B shows continued improvement.
Ablation Study Results
- Warm-up SFT: SFT brought significant performance improvements across all experimental settings (figure (a)). These results show that warm-up SFT accelerated reward improvements and keeps gradient norms low (magnitude of the gradient vector, showing how large the model’s parameter updates are during training) throughout RL training, proving it focuses on improving performance rather than just matching formats when moving from short- to long-context reasoning (figure (b)).
- Curriculum-Guided Staged RL: The phased RL method (figure (a), the green column) led to significant performance improvements. In contrast, single-stage RL showed greater instability, as evidenced by fluctuations in KL divergence and collapse during training (figure (c)).
- Difficulty-Aware Retrospective Sampling: Introducing difficulty-aware retrospective sampling allowed some hard samples from earlier training stages to be retained. When combined with phased RL, this further enhanced performance (figure (a)). Even after phase I RL training, these difficult examples still produce lower rewards and higher policy entropy, encouraging the model to explore more (figure (d)).
Additional Analysis
- The trade-off between SFT and RL in Optimization: Although SFT requires more training data, it still offers significant practical benefits, which include lower computational complexity, minimal infrastructure requirements, and less reliance on specialized technical knowledge. This makes it an economical and efficient strategy for performance improvement.
- SFT and RL serve different but complementary purposes: SFT delivers acceptable performance at a lower cost, while RL is essential for achieving optimal results.
- To maximize performance, RL should build upon SFT: Overemphasizing SFT may lead the model into local optima, thereby limiting RL’s potential for further improvement.
- Emergence and Dynamics of Long-Context Reasoning Behaviors
- Context Grounding: The model retrieves relevant information from a long context to support subsequent reasoning. Example:
Let me look through the provided text to find… - Subgoal Setting: The model decomposes complex problems into smaller subgoals and solves them step-by-step. Example:
To solve this, we first need to… - Backtracking: The model identifies mistakes in its output and revises its reasoning path. Example:
This approach won’t work because… - Verification: The model uses self-reflection to validate its predicted answer. Example:
Let’s verify this result by… - All LRM exhibited significant reasoning behaviors. Among these, long-context grounding occurred most frequently, highlighting its essential role in leveraging long-context dependency during processing.
- RL training gradually strengthens these behaviors and correlates with significant performance gains. This shows that RL effectively guides the model to reason toward correct solutions.
- Although SFT models display some increased reasoning behaviors, this does not consistently transform into performance gains. This may be due to the inherently imitation-based nature of SFT, which tends to focus on surface pattern alignment rather than learning robust reasoning skills.
The figure below shows that:
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...