AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
10.4 Left Padding or Right Padding?
Simply put, in decoder-only models like GPT, Gemma, etc., we use the right padding for training and the left padding for inference.
Left Padding for Inference
let’s start with the simple one. It’s easy to understand why we need left padding for inference. For example:
Let the input be like:
[”I love LLM”, “I love ML and AI”]For the input of
I love LLM, if using the right padding, it becomes I love LLM [pad] [pad]. In such a case, we are basically asking the model to predict the next token based on the [pad] tokens, which is not the correct way.If we use the left padding, then the input becomes
[pad] [pad] I love LLM; the next predicted token will be closely adjacent to the meaningful input tokens.Right Padding for Training
Let’s say we have two samples in a batch:
Input1:
”I love LLM.”; output1: “I love it too!”Input2:
“I love ML and AI.”; output2: “I love them too!”During training, they will be concatenated using the right padding as
You can see that, for the first sample, when the model predicts the output
I love it too!, it’s based on the previous tokens I love LLM. without [pad] in between. Also, we don’t calculate the loss for predicting the [pad] based on I love it too!, so the model won’t expect [pad] always after the end of a sentence.In such a case (right padding during training), we will use left padding during inference to make the prediction pattern consistent.
What if we use left padding for training as well?
In this case, we will get training samples like
Looks not bad, isn’t it? We still keep the sentence coherent without
[pad] in between. And this is more consistent with the inference.But here is the problem: we control that the loss is not calculated on tokens of
[pad], but we cannot control the loss is not calculated on the first meaningful token I based on the previous tokens [pad]. Such additional meaningless loss may interfere with the LLM training and convergence dynamics.What happens if use the right padding during inference?
- Example Code
When padding_side="left", with the input
“1, 2, 3”, we getWhen padding_side="right", with the same input
“1, 2, 3”, we getYou can observe there is an obvious non-consistent between the input and output. This aligns with our assumption that the right padding during inference hurts the model's performance and may lead to unexpected behavior.
Another example: MedGemma
In Google MedGemma’s official fine-tuning code, they explicitly indicate using right padding for training and left padding for inference
- Training Config
- Inference Config
Case Study: GRPO in VLM-R1
GRPO is introduced in 3.4.1 GRPO
When we say right padding for training and left padding for inference, we are talking about the SFT; when it comes to the RLHF, the situation is different. Let's take GRPO as an example using the code from VLM-R1.
Data Preparation for VLM Module
Code from here
Vision language model (VLM) is basically an LLM with an additional vision module to support vision (e.g., image) input. But you can observe here the padding side is set to
left instead of right during training.This is because, in the SFT phase, the LLM is trained with teacher-forcing, which means the model is trained in a parallel behavior by inputting all prompts + ground truth outputs into the model. Like the example we mentioned before:
Where
I love LLM is the prompt and I love it too! is the ground truth output. In this case, we surely use the right padding.But during GRPO training, let’s say we use the outcome reward model (ORM), the outputs are not forced with ground truth text. Instead, we use rule-based reward by extracting the answer from the model’s output and comparing it with the true label, or a reward model (a language model too), or LLM-as-a-judge, i.e., letting an additional LLM judge if the question is correctly answered. In this case, we need the complete output from the trained model, so we are basically doing inference to get it. This is why we prepare the VLM inputs using left padding, making it look like we are using left padding for training.
Reward Model Input Processing
Code from here
As you remember, the reward model is usually frozen during RL like PPO, GRPO, etc. This seems to suggest using left padding because it’s just pure inference.
In fact, the case for the reward model is different again. The reward model is trained to output just a single reward score (in ORM), or we use another LLM (LLM-as-a-judge) to give a reward using prompt engineering.
In the above code, it uses the reward model, which is usually trained with the right padding, i.e., with input
"I love LLM. I love it too! [pad] [pad]", the reward model outputs 0.9 to encourage the LLM output I love it too! based on the user’s input I love LLM.In such a case, the reward model is trained with the right padding, i.e. the padding is part of the inputs for calculating loss, so we should also use the right padding for inference.
But if it’s LLM-as-a-judge, the reward score is acquired by prompt engineering, so there is no usage of left padding or right padding.
Note that, in our example code of 8.4 OpenRLHF: Reward Model Training and Code Analysis, we do similar thing: truncate inputs that are too long and right-pad ones that are too short.
Prev
10.3 KL(P||Q) or KL(Q||P)?
Next
10.5 What is QK-Norm?
Loading...