AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
10.1 Why GRPO training loss starts at 0
To understand why it starts with zero, let’s review the GRPO loss functions. As mentioned in GRPO Loss Analysis, the loss of GRPO is formulated as follows:
Usually, we model the generation process as a single-step MDP and use the outcome reward model (ORM) for learning, so the state is the prompt and the action is the output . In this case, we basically only perform one exploration step per iteration (this is a standard implementation of GRPO, referring to open-r1 and VLM-R1). Therefore,
Then, the in the loss function becomes:
Let’s simplify as . Consequently,
We know that because we are using ORM, and the is a normalized reward, so its expectation across generations is actually zero. Therefore, the loss function can be simplified as
Now, the remaining term is only depending on the current policy and the reference policy . Since these two are initially equal (both the policy and reference models are initialized from the SFT model), the would be zero. This is why the loss starts at zero. As the training goes on, the policy model will be updated and diverge from the reference model, so the loss increases.
The loss will converge to a proper level, where the current policy model is far enough from the reference model to generate more pleasing responses and simultaneously not too far to avoid performance degradation and generating unexpected/unwanted outcomes.
Reference: https://github.com/huggingface/open-r1/issues/239#issuecomment-2646297851, where the is the objective to maximize during training; it’s equivalent to the to minimize during training.
Prev
9.5 LLM-as-Judge
Next
10.2 VLM-R1 Reproduction Results
Loading...