AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression🔗9.1.11 Kimi K2
Kimi K2 is an MoE model oriented toward Agentic Intelligence, with 1T total parameters and 32B active parameters.
Paper link: https://arxiv.org/pdf/2507.20534
Newest Model: https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905
Key Highlights:
- Pre-training
- MuonClip optimizer
- Improving token utility with rephrasing
- Model architecture adopts ultra-sparse MoE (Mixture-of-Experts) and MLA (Multi-head Latent Attention)
- Training infrastructure combines PP (Pipeline Parallelism), EP (Expert Parallelism), DP (Data Parallelism) (ZeRO-1), activation recomputation, FP8 storage, and CPU offloading
- Post-training
- SFT: Agentic data synthesis for tool use learning
- RL:
- Verifiable rewards gym
- Self-critique rubric reward
- Algorithm optimization: added budget control, PTX loss, temperature decay
Benchmark test results:
Introduction
Research Background
- The current development direction of LLM is Agentic Intelligence. It can perform dynamic perception, planning, reasoning, and action in complex environments (i.e., continuous interaction with the environment to acquire new out-of-distribution skills). Compared to traditional RL (For example, PPO for the CartPole game), agent intelligence relies more on the LLM models, where perception and decision-making are integrated. Through its own exploration and exploitation capabilities, agent intelligence can overcome data limitations.
- Pre-training: With limited high-quality data, it is necessary to improve token efficiency (learning signal per token).
- Post-training: Agent data is scarce for tasks like multi-step reasoning, long-term planning, and tool use, so it is necessary to construct scalable, high-quality data synthesis methods.
Research Content
- MuonClip Optimizer: Efficient token utilization with Muon algorithm combined with QK-Clip stability enhancement method.
- Introduction of a large-scale agent data synthesis pipeline, systematically generating demo data for tool use via simulation and real-world environments.
- Designed a framework combining verifiable rewards (RLVR) with self-critique rubric reward mechanisms for RL.
Background - Muon and Its Problem
1. A Quick Recall: SGD Optimizer with Momentum
In standard stochastic gradient descent (SGD), we update parameters directly by subtracting the gradient. To stabilize this process, momentum accumulates a moving average of past gradients:
Here is the current gradient, the momentum coefficient, a weight decay term, and is the weight of the model. This helps smooth noisy updates and encourages consistent movement in productive directions.
2. Muon Optimizer
Reference code: https://github.com/KellerJordan/Muon
Muon (MomentUm Orthogonalized by Newton–Schulz) keeps the momentum buffer , but changes the update rule. Instead of applying the raw momentum directly, Muon orthogonalizes it using a special matrix operation called msign:
Muon normalizes at the matrix level, making the update direction well-conditioned across all modes.
3. How msign Works
Given a momentum matrix , we can take its singular value decomposition (SVD):
The matrix sign is then defined as:
This has two important properties:
- Orthogonal structure: rows/columns are mutually orthogonal, preserving the geometry of the update.
- Equal distance per mode: all singular values are replaced by 1, so no single direction dominates, i.e., the update strength is spread evenly across the active modes.
4. What is MaxLogit, and Attention MaxLogit Explosion in Muon
When training large models with Muon, researchers observed the MaxLogit explosion problem: the maximum attention score (logit) before softmax keeps growing during training.
- In attention, we compute scores as dot products between query () and key () vectors.
- The MaxLogit is simply the largest dot product value among all pairs.
- This is not a common problem across all logits; it usually happens only in one or a few extreme cases, where some queries and keys align too strongly.
7. Existing Attempts for Muon MaxLogit Explosion
Several strategies have been proposed to control the MaxLogit explosion in models trained with Muon:
- Weight Decay: Muon already includes weight decay in its update rule. This helps prevent MaxLogit issues in smaller models. However, in very large-scale models, too much weight decay can harm performance.
- Soft-cap clipping: A direct way to bound logits is to apply a smooth clipping function such as:
This ensures that attention logits remain within limits. But before clipping is applied, the dot products between queries and keys may still grow excessively.
- QK-norm: Another approach is to normalize queries and keys with RMSNorm:
This works well in many architectures, but it does not fit well with MLA (Multi-Head Latent Attention), because query/key normalization can differ between training and inference.
Pre-training
Employ MuonClip Optimizer and QK-norm
Kimi K2 uses the token-efficient Muon optimizer (see previous section). Research from MoonLight shows that under the same amount of training data, Muon significantly outperforms AdamW.
To address the MaxLogit explosion issue of Muon, Kimi K2 proposes QK-Clip to re-adjust the projection weights of Query (Q) and Key (K) after weight updates to limit the growth of attention logits. For each attention head, the maximum logit is defined as the largest input to the softmax in batch B:
The core idea of QK-Clip is:
When exceeds a threshold , the weight matrices of and are clipped. Since in practice only a small subset of heads will experience logit explosion, to minimize the impact on model training, head-wise QK-clip is applied as follows:
where
In short, QK-Clip dynamically rescales and weights per head when attention logits get too large, preventing instability while keeping most heads unaffected. The above operation can be directly applied to the MHA (Multi-Head Attention). For MLA, however, QK-Clip only applies to non-shared heads.
Here, we can revisit how MLA handles RoPE (see Section 1.3.5 MHA to MLA - Multi-Head Latent Attention (MLA)). The solution in MLA is to combine extra multi-head queries and shared keys with RoPE.
- and : Each is scaled by .
- (extra multi-head queries): Scaled by .
- (extra shared key): Remains unchanged, to avoid affecting other attention heads.
Improving Token Utility with Rephrasing
By leveraging domain-specific rephrasing techniques (e.g., rewriting in styles of knowledge domains, multi-perspective rephrasing, or note-style rewrites in mathematics), Kimi K2 improves the utilization rate of high-quality data and reduces overfitting.
- Token Efficiency: The performance gain achieved per token consumed during training. A simple way to increase token efficiency is to let the model repeatedly encounter the same tokens, but this may cause overfitting and reduced generalization ability.
- Kimi K2 mainly focuses on rephrasing techniques for knowledge-domain data and mathematics-domain data.
Knowledge Data Rephrasing
Knowledge-dense datasets require multiple rounds of rephrasing training so that the model can fully absorb the knowledge. However, this reduces token efficiency and increases the risk of overfitting. To address this, K2 adopts the following synthetic data framework:
- Style- and Perspective-diverse Prompts: A series of carefully designed prompts guides the model to generate rephrasing in different styles and perspectives.
- Chunk-wise Autoregressive Generation: The text is split into multiple chunks, each re-described separately, and then concatenated together, which helps mitigate hidden length constraints.
- Fidelity Verification: Each rephrasing segment is checked against the semantics of the original text to ensure alignment, providing initial quality control.
Mathematics Data Rephrasing
Reference: SwallowMath https://arxiv.org/abs/2505.02881
- High-quality mathematics documents are rewritten into a “learning notes” style, enhancing readability and diversity.
- High-quality mathematical resources in other languages are translated into English to further increase dataset diversity.
This is similar to the approach in Textbook is All You Need, where raw data is rewritten into high-quality textbook-style case-study data.
Model Architecture
By adopting MoE and MLA, compared with DeepSeek-V3, Kimi K2:
- Increases the number of experts (384 total, with 8 active per token).
- Reduces the number of attention heads.
- Achieves a better balance between performance and inference efficiency.
Post-training
SFT: Agentic Data Synthesis for Tool Use Learning
Two core principles: maximize prompt diversity and ensure high response quality. The focus is to design agent-oriented data by building a data synthesis pipeline. This pipeline simulates large-scale real-world tool-use scenarios, enabling the model to learn tool-use capabilities through multi-step interactive reasoning. It mainly consists of three stages:
1. Tool Specification Generation
The first step is constructing a large tool repository, which includes both real-world tools and LLM-synthesized tools.
- From GitHub repositories: Directly collected 3,000+ real MCP (Model Context Protocol) tools.
- Through hierarchical domain synthesis: Starting from core categories (e.g., financial trading, software applications, robotics control, etc.), multiple subdomains are derived within each category. For each subdomain, specialized tools are synthesized with clear interfaces, descriptions, and operation definitions, resulting in 20,000+ synthetic tools.
2. Agent and Task Generation
For each toolset, generate an agent equipped with that toolset along with the corresponding tasks.
- Synthesize diverse system prompts and assign each agent a different toolset combination, producing thousands of agents. Each agent is equipped with different capabilities, domains of expertise, and behavioral modes, ensuring broad coverage of potential tool-use scenarios.
- Task generation based on evaluation standards: For each agent configuration, generate a wide range of tasks that range from simple to complex operations. Each task has clear evaluation criteria, including success standards, expected tool-use patterns, and checkpoints for assessment. This guarantees that the evaluation of agent capabilities remains consistent and objective.
3. Trajectory Generation
For each agent and its tasks, generate the interaction trajectories of agents completing tasks by invoking tools.
- User Simulation: The LLM generates user roles with unique communication styles and preferences. These simulated users engage in multi-turn dialogues with agents, constructing realistic interaction scenarios.
- Tool Execution Environment: A complex tool simulator (functionally equivalent to a world model) executes tool calls and provides realistic feedback. After each execution, the simulator maintains and updates state, supporting complex multi-step interactions with persistent effects. Controlled randomness is introduced, producing a variety of outcomes including success, partial failure, and edge-case scenarios.
Additional
- Quality Validation and Filtering
- Using the LLM-as-a-Judge method, each trajectory is evaluated against task evaluation standards.
- Only trajectories that meet the success criteria are retained for training.
- Real Execution Enhancement
- For scenarios where realism is especially critical (e.g., coding and software engineering tasks), the simulation environment is combined with real sandboxes.
- This allows execution of actual code, interaction with real development environments, and provision of realistic feedback.
Reinforcement Learning
A Gym-like (remember that in Section 3.2 Reinforcement Learning we use gym for environment simulation), extensible framework is developed that facilitates RL across a wide range of scenarios.
1. Mathematics, STEM, and Logic Tasks
- Diversity Coverage: For mathematics and STEM tasks, combine expert annotations, internal QA pipelines, and open dataset collection to obtain high-quality QA pairs. Additional coverage for under-covered domains is introduced through tagging systems. Logic task datasets include structured data tasks (e.g., multi-hop reasoning, cross-table aggregation) and logic puzzles (e.g., 24-game, Sudoku, etc.).
- Moderate Difficulty: Based on the SFT model pass@k accuracy, task difficulty is assessed. Only medium-difficulty problems are selected as RL prompts, avoiding overly simple or overly difficult problems that would weaken the learning signal and reduce training efficiency.
2. Complex Instruction Following
- Hybrid Verification Framework:
- Verifiable outputs are evaluated deterministically via code interpreters.
- Fine-grained, reasoning-intensive instructions are assessed by LLM-as-judge.
- Additional hack-check layers are introduced to detect deceptive model behaviors.
- Multi-source Instruction Generation: Training data is built using three generation strategies, which ensure breadth and depth in instruction coverage.:
- Carefully designed complex conditional prompts and evaluation standards from the data team.
- AutoIF-inspired intelligent instruction augmentation, generating broader instruction variations.
- Expert-crafted instructions targeting specific failure modes or edge-case scenarios for fine-tuning.
3. Faithfulness
- Inspired by the FACTS Grounding evaluation framework, sentence-level faithfulness evaluation models are trained for automatic verification.
- These models detect unsupported or hallucinated statements within context, filtering them out and serving as reward models to improve overall faithfulness performance.
4. Coding and Software Engineering
- Data sources: Tasks are collected from open-source datasets and synthetic sources, along with evaluation standards. High-quality human-written unit tests (from pre-training data) are integrated to ensure data diversity and reward signal accuracy.
- Software engineering tasks: Large numbers of pull requests and issues are gathered from GitHub. A sandbox-based development environment is built to support the execution of user prompts/issues and unit tests. This environment leverages powerful sandbox infrastructure for reliable testing.
5. Safety
- Prompt Collection: Begin with carefully designed seed prompts covering common risk categories such as violence, fraud, and discrimination.
- Adversarial Simulation: Through automated pipelines combining:
- Attack models: generate adversarial prompts to exploit potential vulnerabilities in the target LLM.
- Target models: LLM under test, responding to adversarial prompts.
- Judge models: judge whether adversarial prompts succeeded in bypassing safety mechanisms.
The system simulates complex jailbreak attempts. Using specific task evaluation standards, each interaction is assessed, and the evaluation model outputs binary labels of success/failure.
Self-Critique Rubric Reward
Introduces a general RL framework based on self-evaluation feedback, where the model generates its own preference signals by evaluating its own outputs. By expanding capabilities learned in verifiable environments into broader subjective tasks, the LLM is aligned with subtle human preferences, including helpfulness, creativity, depth of reasoning, factuality, and safety.
1. Self-Critiqued Policy Optimization - For Actor
- In the first core loop of learning, K2 acts as the actor, generating responses to general prompts that cover a wide range of application scenarios.
- Then, K2 acts as the critic, ranking all outputs pairwise according to multiple self-evaluation standards. These standards include:
- Core criteria that reflect Kimi’s fundamental values,
- Prescriptive standards such as eliminating reward hacking behavior,
- Human-annotated standards are specifically designed for certain instructions or scenarios.
Although some standards can be mandatory, K2 maintains flexibility based on internal empirical balancing.
This allows the model to continuously adapt its self-evaluation strategies while evolving, ensuring that alignment with specific instruction responses remains consistent with its core identity.
2. Closed-Loop Critic Refinement and Alignment - For Critic
During RL training, the critic model is optimized using verifiable reward signals: On-policy trajectories generated based on verifiable reward prompts are used to update the critic. This step integrates objective verifiable signals from RLVR into the critic model.
For complex and ambiguous reward-signal tasks, subjective evaluations are built on top of verifiable data, allowing critic models to improve their ability to evaluate reasoning reliability in verifiable tasks.
This iterative loop ensures that critic evaluation strategies evolve dynamically and adjust synchronously with standards. Ultimately, this enables stable and scalable alignment of models with both complex verifiable tasks and non-verifiable human objectives.
RL Algorithm
Based on the strategy optimization algorithm proposed in K1.5 (Section 9.1.2 Kimi-K1.5), for each problem , sample responses from the previous policy , and optimize the model using the following objective function:
Algorithm Optimization Strategies
- Budget Control: Improves Token Efficiency.
To address RL’s common issue of overly long generations and increased inference costs in non-reasoning domains, a maximum token budget per task type is set during training. Any response exceeding the budget is truncated and penalized, encouraging the model to generate concise and effective solutions within limits. This Significantly
- PTX Loss: Improves the Model’s Generalization and Diversity
To prevent RL training from forgetting high-quality data, a curated dataset of high-quality samples is maintained. The PTX (Pre-Training Cross-Entropy Loss) loss is incorporated into the RL objective, which leverages the strengths of high-quality data while reducing overfitting to limited task distributions.
- Temperature Annealing: Balances creativity, stability, and quality.
For tasks involving creative writing or complex reasoning, training initially uses a high sampling temperature to encourage exploration and diverse outputs. Later, the temperature is gradually reduced, shifting exploration into exploitation to stabilize outputs while maintaining quality.
Evaluation
Kimi K2 0905
This new model from Moonshot AI offers several key improvements:
- Increased Context Length: The context window has been expanded from 128K tokens to 256K tokens, allowing it to handle much longer documents and conversations.
- Enhanced Agentic Programming: Its performance on public benchmarks and in real-world agent programming tasks has been significantly improved.
- Better Frontend Coding: The model now produces more aesthetically pleasing and practical output for frontend development.
Prev
CivitAI’s Payment Issue
Next
Awesome-AI-Tutorials
Loading...