AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression🔗9.1.6 Seed-Thinking-v1.5
Seed-Thinking-v1.5 is a 200B-parameter MoE model with 20B active parameters. Its benchmark performance includes:
- AIME 2024: 86.7
- Codeforces: 55.0
- GPQA: 77.3
It demonstrates outstanding reasoning abilities in STEM and programming domains. On non-reasoning tasks, it outperforms DeepSeek R1 by 8%, indicating a broader application scope.
Introduction
Models like o1, R1, Gemini 2.5 Pro, and Claude 3.7 have shown significant advancements in reasoning, mathematics, and programming tasks. The development of reasoning models is shifting towards being more structured, efficient, and scalable. This research focuses on improving training efficiency, long-chain thinking, and large-scale reinforcement learning (RL). This paper devotes significant effort to three key points: training data, RL algorithms, and RL infrastructure:
- Training Data:
- Too much non-reasoning SFT data significantly reduces the model’s exploratory ability.
- RL data: STEM problems, code-related tasks, logical reasoning, and creative writing/dialogue as non-reasoning data. Among these, logical reasoning data has made a significant contribution to performance on the ARC-AGI benchmark, with math data showing strong generalization capabilities, boosting performance across various tasks.
- RL Algorithm: RL training for reasoning models is highly unstable and prone to interruption, especially for models that have not gone through SFT. Seed developed two stable RL algorithms: DAPO and VAPO (see Section 3.4.5 DAPO and Section 3.4.6 VAPO).
- RL Infrastructure: To handle the workload of distributed training, this study decouples streaming rollout architecture, with prioritized sample pools for asynchronous trajectory generation. This achieves 3x faster iteration speed than synchronous frameworks and supports automatic fault recovery for mixed-precision fine-tuning.
RL Training Data
Reinforcement learning training data mainly consists of two parts: verifiable questions with definite answers and non-verifiable questions without definite answers. The model’s reasoning ability mainly originates from the first part and can be generalized to the second part.
Verifiable Questions
- STEM Questions with Definite Answers
- Data Source: 100K high-quality competition-level questions, covering mathematics (over 80%), physics, and chemistry, sourced from open datasets, public competitions, and private question banks.
- Data Cleaning:
- First, remove questions with incomplete statements, inconsistent notation, or unclear requirements.
- For multiple-choice questions, multiple answers are generated using Doubao-1.5 Pro. If the model scores 1 on woN (Worst of N) for a question (run the model N times, and even the lowest score is 1), then that question is considered too simple and is removed.
- Some reference answers may be inaccurate. The SOTA reasoning model is used to generate multiple candidate answers. If the model's answer differs from the reference but shows high internal consistency or contains only a small number of reasoning tokens, the reference is flagged as incorrect. Human experts then manually verify these questions to ensure correctness.
- Using data augmentation to make data more suitable for training and evaluation. Specifically, multiple-choice questions are converted into fill-in-the-blank or short-answer format, to eliminate guessing. Modify some math questions to ensure the reference answer is unique and complete.
- Code Problems with Unit Tests
- Data Source: Selected high-quality and challenging algorithmic tasks, mainly from well-regarded programming competitions.
- Data Filtering: The data is filtered to ensure each problem includes a clear problem description, a set of unit tests, and a checker script:
- Unit tests are used to verify the functional correctness of the solution.
- Checker scripts verify additional constraints like output format and edge cases.
- Data Evaluation: A set of offline evaluation mechanisms has been developed to validate locally. All training and evaluation problems are integrated into an internal code sandbox environment, enabling direct execution and assessment of model-generated code.
Problems are also filtered by difficulty to ensure appropriate complexity and real-world applicability in algorithmic reasoning.
- Automatically Verifiable Logical Reasoning Data. Specifically:
- Task Collection: collected 22 common logical reasoning tasks, such as the 24-point game, mazes, Sudoku, etc. For each task, a data generator and an answer validator were built.
- Data Generation: The data generator can automatically produce large volumes of training and evaluation data, with configurable difficulty levels. During training, difficulty is adjusted gradually based on the model's performance on specific tasks.
- Answer Validation: The answer validator rigorously checks the correctness of generated results, and can be seamlessly integrated into the RL loop as a reward function.
The RL training used 10K logical reasoning puzzle data.
Non-Verifiable Questions
- Data Source: These questions mainly involve non-reasoning tasks that require human preference for quality evaluation, such as creative writing, translation, Q&A, and role-playing tasks. The prompts in this dataset are drawn from Doubao-1.5 Pro's RL training data.
- Data Filtering: Samples with low variance in scores or low difficulty are removed. Specifically, an SFT model generates multiple responses for each prompt, which are then scored by a Reward Model:
- Prompts with low variance in scores are deleted because they have limited diversity in responses and minimal improvement potential.
- During Doubao-1.5 Pro RL training, prompts with reward scores above a certain threshold are also removed, this data may be too simple or already overrepresented in the dataset. Over-optimization on such samples leads to premature convergence and reduces model performance.
- Data Evaluation: A pair-wise rewarding method is used to score and conduct RL training, which compares two samples in terms of quality. This helps the model better understand user preferences and improves both the quality and diversity of its outputs.
Reward Modeling
Reward Modeling for Verifiable Questions
- Seed-Verifier: Based on human-designed principles, a LLM directly evaluates the triplet
<question, reference answer, model-generated answer>. If the reference answer and the model-generated answer are largely equivalent in validity/meaning, the LLM returns YES, otherwise NO.
Note: this doesn't require exact string matches, but instead uses mathematical rules and logic to judge if the two answers are mathematically equivalent.
- Seed-Thinking-Verifier: Trains a verifier to produce fine-grained logical reasoning paths for evaluation, allowing detailed judgments about the similarities and differences between reference and model-generated answers. For more details, please view Section 9.1.4 Deepseek-GRM
Seed-Thinking-Verifier Addresses Three Key Issues of Seed-Verifier in Verifiable Questions
- Reward Hacking: Without a proper "thinking" process, models may exploit loopholes to obtain high rewards for answers that don't solve the problem. By adding detailed logical reasoning steps, reward hacking can be effectively mitigated.
You can review this in Section 3.3: Reinforcement Learning from Human Feedback
- Prediction Uncertainty: For example, with answers like or 524288, although they are mathematically equivalent, the format is different. Seed-Verifier is likely to return YES or NO inconsistently, but Seed-Thinking-Verifier provides much more consistent results.
- Corner Case Handling: Seed-Verifier struggles with complex edge cases, as shown in the example below. In contrast, the Seed-Thinking-Verifier delivers accurate judgments through step-by-step reasoning.
Reward Modeling for Non-Verifiable Questions
A reward model was trained using human preference data from Doubao-1.5 Pro, primarily covering tasks such as creative writing and summarization.
- The reward model uses a pair-wise generative RM format (see also Section 9.1.4 Deepseek-GRM).
- The model evaluates the relative quality of two answers and uses the probability of choosing YES or NO as the final reward score.
This approach enables the model to directly compare differences between responses during evaluation, thus avoiding over-focusing on irrelevant details and improving the stability of RL training.
Approach
SFT Stage: Compared to directly fine-tuning from the base model using RL, SFT models can provide more readable outputs with fewer hallucinations. The SFT stage used 300K verifiable data and 100K non-verifiable data.
Workflow for Generating High-Quality LongCoT Data
- Cold Start: Human experts create diverse reasoning examples by either carefully designing prompts or having interactive dialogues with the model to guide its thinking. After accumulating tens of high-quality cold-start samples, a basic LongCoT reasoning model can be trained.
- Rejection Sampling: Use Seed-Verifier to perform rejection sampling, including for math, code, and other domains.
RL Stage: The data used during the RL stage mainly falls into three categories:
- Verifiable Data: Obtains feedback from the Verifier. This type of data can be directly validated using the known criteria.
- General Data: Scored by the Reward Model.
- Special Data: Scored using both the Verifier and Reward Model.
The specific RL algorithms are primarily based on the previously developed DAPO and VAPO algorithms (see Section 3.4.5 DAPO and Section 3.4.6 VAPO).
Online Data Distribution Adaption
When merging data from different domains, differences in difficulty, the risk of reward-hacking, and other underlying factors, make it hard to improve performance uniformly. To address this, the paper introduces Online Data Distribution Adaption, which transforms fixed RL prompts distributions into adaptive ones (equivalent to a dynamic RL environment). This helps:
- Minimize the negative impact of data interference
- Ensure more balanced improvements across different skill areas
Infrastructures
HybridFlow Framework
Built using HybridFlow, the training workload runs on a Ray cluster. The Dataloader and RL algorithms are implemented within a single process Ray Actor, while model training and response generation (Rollout) are executed on the Ray Worker Group.
Streaming Rollout System
In the Long-CoT generation process, large variation in prompt response lengths causes severe long-tail effects, resulting in significant GPU idling during generation.
The Streaming Rollout System (SRS) architecture decouples model evolution from runtime execution, using a dynamic parameter to adjust the ratio of on-policy and off-policy samples:
- The completion ratio , which represents the proportion of samples generated on-policy using the latest model.
- The remaining non-complete part is allocated to versioned model snapshots for off-policy rollout. These are asynchronously combined using independent resources for seamless integration.
Training System
- Parallel Mechanism: Combines Tensor Parallelism (TP) / Expert Parallelism (EP) / Context Parallelism (CP) with Fully Sharded Data Parallel (FSDP). For training Seed-Thinking-v1.5, TP/CP is used at the attention layer, and EP is applied in the MoE layers.
- Sequence Length Balancing: In Data Parallel (DP) processes, sequence lengths may vary, leading to unbalanced compute loads and low training efficiency. The KARP algorithm is used to rearrange input sequences within each mini-batch to balance them.
- Memory Optimization: Through layer-wise recomputation, activation offloading, and optimizer offloading.
- Auto Parallelism: An AutoTuner is developed that builds performance-aware memory models to estimate the best configuration.
- Checkpointing: Uses ByteCheckpoint to support minimal overhead recovery checkpoints from different distributed setups.
Experiments
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...