AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression🔗9.1.2 Kimi k1.5
This study mainly includes two multimodal reasoning models: Kimi k1.5 LongCoT and Kimi k1.5 ShortCoT, focusing on RL training techniques, multi-modal data recipes, and infrastructure optimization.
- Long-context scaling and improved policy optimization methods are the keys, to eliminating the reliance on complex techniques such as MCTS (Monte Carlo tree search), Value Model, and PRM (process reward models).
- k1.5 LongCoT achieves performance comparable to OpenAI's o1 on multiple benchmarks.
- Through an efficient long2short method, k1.5 ShortCoT outperforms GPT-4o and Claude Sonnet 3.5 on benchmark reasoning tasks.
Introduction
- Research Background: Traditional scaling laws based on the next token prediction are limited by the availability of high-quality training data. This technical report explores a new axis for scaling. Using RL with LLMs, the models learn to explore with rewards and thus are not limited to a pre-existing static dataset.
- Key Ingredients:
- Long Context Scaling: Scale the context window of RL to 128K, with performance continuing to improve as the context length increases.
- Improved Policy Optimization: Derive a formulation of RL for long-CoT and employ a variant of Online Mirror Descent for robust policy optimization, as well as an effective sampling strategy, length penalty, and multi-modal data recipe.
- The approach combines long-context scaling and advanced policy training to produce CoT behaviors, such as planning, reflection, and correction, without relying on complex tools like MCTS, Value Model, or PRM.
- Multi-modalities: k1.5 is a multimodal model trained with RL on multimodal (text and vision) reasoning data.
- long2short: Transfer/distillation of LongCoT capabilities into the ShortCoT model through various methods.
- Research Contribution: RL-based context length scaling is crucial for the sustained improvement of LLMs.
<ins/>
Method
Stage 1: RL Prompt Set Selection
The quality and diversity of the reinforcement learning (RL) prompt set are crucial to successful training. A well-designed prompt set helps guide the model toward robust reasoning while also preventing reward hacking and overfitting to certain answer patterns.
- Diverse Coverage: Prompts span a wide range of domains, including STEM subjects, coding, and general reasoning.
- A filtering process is used to automatically select questions that feature rich reasoning content and are easy to evaluate. This includes both text-only and image-text QA data.
- A tagging system was used to categorize prompts by domain and discipline (M. Li et al. 2023; W. Liu et al. 2023)
- Balanced Difficulty: The prompt set is designed to include a spectrum of difficulty levels (from easy to hard) to support progressive learning and reduce the risk of overfitting to particular complexity tiers.
- Using an SFT model to sample prompts multiple times and calculate pass rate as a difficulty label for prompts.
- Accurate Evaluability: Prompts should encourage the model to follow valid reasoning processes, rather than rely on pattern matching or guessing.
- Some complex reasoning questions may have an easy-to-guess answer. Models may pass without reasoning correctly.
- Questions that are easy to guess are filtered out: multiple-choice, true/false, or proof-based questions.
- Test if a question is easy to hack: prompt a model to guess potential answers without any CoT reasoning steps. If the model predicts the correct answer within N attempts, the prompt is considered an easy-to-hack question and removed. This study found that setting N = 8 can remove the majority of easy-to-hack prompts.
Stage 2: Long-CoT SFT
- Based on the RL prompt set in Stage 1, this study uses prompt engineering and rejection sampling to construct a high-quality long-CoT warmup dataset. This dataset contains:
- Reasoning paths for both text and image.
- Key cognitive processes such as planning, evaluation, reflection, and exploration.
- The base model was applied with SFT using this dataset, enabling it to generate more detailed step-by-step responses.
Stage 3: Reinforcement Learning
- Problem Setting:
- Training dataset with the problem and the ground truth answer
- The intermediate reasoning steps are
- The final answer is
- Verifiable questions: Use rule-based methods, such as math/code-related tasks.
- Problems with free-form ground truth: Train a reward model to predict whether the model's answer matches the ground truth.
Suppose:
The objective is to maximize the expected reward:
The reward is a value of 0 or 1 (not between) given by the reward model. The policy is rewarded with a 1 if its final answer matches the ground truth (or passes the verification test) and with a 0 otherwise. There are no intermediate reward values in this setup. Similar to DeepSeek-R1, the reward depends on the input problem:
A major difference from simple prompt-based approaches is that the model is expected to do more than just execute a sequence of reasoning steps. It should also develop essential planning capabilities (such as error identification, backtracking, and solution refinement) by utilizing the full set of explored thoughts as contextual input.
- Policy Optimization: A variant of Online Policy Mirror Descent is used. Basically, it is a variant of the policy gradient method, plus a KL divergence to constrain the policy (or actor) model.
This study uses an off-policy RL sampling method. It samples from the reference policy. In other words, the response (including the answer and the reasoning steps ) is generated by the reference model and used to update the current policy model (Section 8.5.3 OpenRLHF PPO Training - PPO Training: Simplified Pseudo Code shows why generating the response and calculating the log probability can use different models).
Note that k responses are sampled for each problem. The mean sampled reward is used for value estimation. So, it doesn’t need to train an additional Value Model.
Honestly, quite similar to what DeepSeek-R1 does.
Training Conclusion: The model should be encouraged to explore different reasoning paths to enhance its ability to solve complex problems. The primary goal is not just achieving high accuracy on training questions, but rather to ensure the model learns effective problem-solving strategies.
- Length Penalty: During training, the model tends to overthink, which is costly during training and inference, and not preferred by humans. To address this, a length penalty reward is introduced: for each prompt, multiple answers are sampled, and the and are calculated. The reward for each answer is penalized based on its length.
In essence, shorter responses are preferred and penalize longer responses. If one response has the longest length in the group, the length reward for this response is -0.5.
- Sampling Strategies: Although RL algorithms inherently have relatively good sampling properties (with more difficult problems providing larger gradients), their training efficiency is limited. Additional sampling strategies are used:
- Curriculum Sampling: Start by training on easier tasks and gradually progress to more challenging ones.
- Prioritized Sampling: Problems with lower success rates receive higher sampling probabilities
- Additional Details:
- Test Case Generation for Coding: Using CYaRon and the base Kimi k1.5 model to generate test cases for the coding problem to train the model with RL.
- Reward Modeling for Math: Math problems often have different forms of correct answers (e.g., fractions vs. decimals). This study adopted two methods to improve the reward model’s scoring accuracy:
- Classic RM: A value-head-based reward model. The model takes the
question, thereference answer, and theresponseas input, and outputs a single scalar score representing the correctness of the response. - Chain-of-Thought RM: A reward model trained on CoT data and generates reasoning steps before providing a final correctness judgment.
- Vision Data: Includes real-world data, synthetic visual reasoning data, and text-rendered data.
- Real-world data covers a wide range of scientific questions across various grade levels, requiring graphical comprehension and reasoning, location guessing, and data analysis.
- Synthetic visual reasoning data is artificially produced, including procedurally generated images and scenes designed to strengthen targeted visual reasoning abilities, such as interpreting spatial relations, recognizing geometric patterns, and analyzing object interactions.
- Text-rendered data involves converting text (e.g., documents, code, or structured data) into images (e.g., screenshots). This enhances the model’s consistency on text-visual tasks and improves performance on text-heavy images.
Both methods were applied to 800K data. Ultimately, CoT RM achieved 14% higher accuracy than traditional RM.
Long2Short
To transfer/distill the capabilities of the LongCoT model into a ShortCoT model, the following strategies are applied:
- Model Merging: averaging the weights of the LongCoT and ShortCoT models.
- Shortest Rejection Sampling: Multiple responses are generated, and the shortest correct one is chosen to train the smaller model via SFT. This addresses the variability in response length.
- DPO: Similar in approach to Shortest Rejection Sampling. The shortest correct answer is used as the preferred output, while longer responses (including both correct and incorrect) are treated as rejected samples for DPO training.
- Long2Short RL: Following standard RL training, the model that best balances performance and token efficiency is selected. This model is then used in a separate Long2Short RL phase, where a length penalty is applied to the reward, and the maximum rollout length is substantially reduced to encourage more concise outputs.
Other Training Details
The k1.5 model is trained by Pretrain, SFT, and finally RL.
- Pretrain: Involves three stages:
- Vision-language pretraining to establish a solid language foundation, followed by gradual multimodal integration.
- Cooldown brings together skills using carefully chosen and synthesized data, especially for tasks that involve reasoning and knowledge.
- Long-context activation, extending the sequence processing up to 131,072 tokens.
- Vanilla SFT: Standard SFT on a dataset containing about 1 million text examples. Specifically:
- 500k for general QA
- 200k for coding
- 200k for math and science
- 5k for creative writing
- 20k for long-context tasks (summarization, document QA, translation, and writing)
- 1 million text-vision examples covering a wide variety of types, including:
- Chart interpretation
- OCR
- Image-grounded conversations
- Visual coding
- Visual reasoning
- Math/science problems with visual aids
RL Infrastructure
- Large Scale RL Training System for LLM
- Master node: Manages communication and data transfer between the rollout workers (for inference/sampling), trainer workers, evaluation with reward models, and the replay buffer.
- Trainer: Compute gradient updates.
- Rollout workers: Generate rollout trajectories and produce sequences of responses for the input prompt. These trajectories are then stored in a replay buffer. Periodically update the rollout worker’s weight using the latest Policy model.
- Partial Rollouts for Long CoT RL: Helps handle long-context RL training.
- A fixed output token budget is set, limiting how many tokens can be generated.
- Unfinished rollouts (if the task exceeds the token budget) are stored in the replay buffer and continued in the next training iteration. This avoids wasting system resources on overly long trajectories.
- Because rollout workers operate asynchronously, when some are engaged with long trajectories, others can independently process new, shorter rollout tasks.
- Repeat detection: repeated sequences are also detected and terminated early to reduce unnecessary computation while maintaining output quality
- Hybrid Deployment of Training and Inference
- Training Phase: Initially, Megatron and vLLM run in separate containers, managed by a shim process called checkpoint-engine. Megatron begins the training. After training is completed, it offloads GPU memory and prepares to transfer the current weights to vLLM.
- Inference Phase: Once Megatron finishes unloading, vLLM starts with dummy weights and receives the latest model weights from Megatron via Mooncake. After the rollout process is complete, the checkpoint-engine stops all vLLM processes.
- Subsequent Training Phase: When vLLM’s memory is freed, Megatron reloads its memory and resumes the next round of training.
Mooncake is open-sourced by Moonshot AI
- Code Sandbox: A sandbox is developed as a secure environment for executing user-submitted code, optimized for code execution and code benchmark evaluation
<ins/>
Experiment
Evaluation
Benchmarks:
- Text Benchmark: MMLU, IF-Eval, CLUEWSC, C-EVAL
- Reasoning Benchmark: HumanEval-Mul, LiveCodeBench, Codeforces, AIME 2024, MATH500
- Vision Benchmark: MMMU, MATH-Vision, MathVista
For K1.5 long-CoT model:
For the K1.5 short-CoT model
Ablation Studies
- Scaling of model size and context length: The larger model generally shows better token efficiency than the smaller model. This suggests that, for maximizing performance, increasing the context length of a larger model offers both a higher performance ceiling and better efficiency. However, when test-time computing is constrained, training smaller models with extended context lengths can be a good alternative.
- Effects of using negative gradients: Compared with the Reinforced Self-Training (ReST) algorithm that selects only the best response, this approach demonstrates that incorporating negative sampling significantly improves the efficiency of generating CoT (Chain of Thought) reasoning. This method not only enhances reasoning quality but also optimizes the training process, achieving stable performance with fewer training samples.
In the this study, "negative gradient" refers to penalizing incorrect outputs during policy optimization. Unlike ReST, which only reinforces good samples (positive gradients), Kimi's method also applies negative gradients to reduce the likelihood of generating poor responses.
This is pretty similar to GRPO, where both good and bad samples generated within the same group contribute to the final loss and gradients.
- Sampling strategies: The Curriculum learning approach is better than uniform sampling.
Conclusion
- Scaling of context length and improved policy optimization are key, removing the need for MCTS, Value Model, and PRM.
- A Long CoT RL framework was established, using a variant of Online Mirror Descent for effective policy optimization.
- Through an efficient long2short method, the k1.5 ShortCoT model achieved better benchmark reasoning results than GPT-4o and Claude Sonnet 3.5.
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...