AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
10.3 KL(P||Q) or KL(Q||P)?
ㅤ | Format | Penalty | Usage |
Forward KL | The true distribution is P, and we use Q to approximate it. It penalizes places where P has probability but Q does not. | Variational Inference, mass-covering, etc. | |
Reverse KL | The true distribution is Q, and we use P to approximate it. It penalizes places where Q has probability but P does not. | GANs, mode-seeking, etc. |
We talked about the mass-covering in 3.5.7 TDPO Algorithm as well.
In the case of training a machine learning model (e.g., DL, LLM, etc.), let’s say we have as the real distribution and as the distribution that approximates the real one, where the is parameterized by the model parameters and we have as .
- Forward KL (mass-covering):
This KL encourages to cover all possible regions of . would rather be too broad than miss any part of . Therefore, this KL is adopted by TDPO to improve the diversity of generated responses. It is also suitable for variational inference because we prefer to cover rather than miss key areas.
- Reverse KL (mode-seeking):
This KL encourages to focus on the most probable regions, ignoring uncommon areas. This is commonly used in maximum likelihood estimation (MLE) or generative adversarial networks (GANs), as it tends to find the most likely explanation. It is also why GANs tend to mode collapse, which occurs when the generator produces a limited set of outputs, lacking the diversity expected from the training data distribution.
Example Code
WHY Forward KL usually has a larger loss than Reverse KL?
- Forward KL is mass-covering, penalizing regions where Q fails to cover P
Mathematically:
When but :
So the KL value becomes very large. In other words, Forward KL is very sensitive to missing probability mass, especially when Q fails to cover the tail or modes of P, resulting in very large penalties.
- Reverse KL is mode-seeking, more likely to "avoid" difficult regions
Mathematically:
When but : There’s a large penalty, but during optimization, Q can actively avoid such regions (because it decides where to place its mass).
So during optimization, Reverse KL tends to greedily choose one mode, ignoring the full support of P, which can make the loss appear small.
Prev
10.2 VLM-R1 Reproduction Results
Next
10.4 Left Padding or Right Padding?
Loading...