AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
10.6 Why LLM training rarely adopts Dropout?
The reason for not using Dropout during LLM training is not about LLM, but it’s about regression tasks. Let’s start with analyzing what the Dropout is doing.
1. Dropout Formula
Here we analyze how dropout affects the statistical properties of a neuron's output. We focus on the common implementation known as
"Inverted Dropout".Let
h be the original output of a neuron. Let p be the dropout probability (the probability of setting the output to zero).With inverted dropout, the output
h′ after dropout is defined as:The scaling by
1−p is a step that ensures the expected value of the output remains the same, which helps keep the network's behavior consistent between training and inference.To analyze this, we can model the process using a Bernoulli random variable
d, where:Then, the output
h′ can be written as:2. Dropout Mean
Let's find the expected value of the output
h′, denoted as E[h′]. We assume that the dropout mask d and the neuron's output h are independent. Let .Leveraging the independence of
d and h:The expected value of the Bernoulli variable
d is . Substituting this in:Clearly, The mean of the neuron's output does not change after applying the inverted dropout.
3. Dropout Variance
Now, let's compute the variance of
h′, denoted as Var(h′). Let .The formula for variance is . So we have
We already know , so . Next, we compute :
Still assuming independence, . Then for a Bernoulli variable
d, (since and ). Therefore, . We also know that , which means .Substituting these results back into the equation for :
Finally, we can calculate the variance of
h′:We can simplify this expression as:
Therefore, the variance of the output increases. The new variance depends on the original variance , the original mean , and the dropout probability
p.4. Why Dropout May Hurt Performance in Regression Tasks
After the Dropout layer, the output is passed through a non-linear layer (e.g., Linear + ReLU). Non-linear layers transform the input distribution. For example, ReLU sets all values below 0 to 0.
This transformation means that the output mean of the non-linear layer becomes a function of both the mean and variance of the Dropout output.
Although Dropout preserves the mean during inference (by scaling), it changes the variance. This variance shift causes the output mean of the non-linear layer to shift, leading to biased final predictions.
Impact on different tasks
- In regression tasks, this output shift can significantly affect performance since the model outputs absolute values.
- In classification tasks, the model outputs logits (e.g., 0 or 1), so it's more robust to such variance-induced shifts.
In Conclusion, when we use Dropout, we are basically balancing between the pros of preventing overfitting brought by Dropout and the cons of distribution shifts brought still by Dropout.
It’s just because the classification tasks are more robust to such cons, so the pros win, and we get what we want, but we don’t get the same on regression tasks because of the nature of
“regression”.Is LLM Training a Regression Task?
Let’s take pre-training as an example, where a teacher-forcing strategy is applied. In this case, we are using cross-entropy loss and softmax, which makes it look like a classification task in each step, i.e., based on previous tokens, choosing the next token.
But unlike traditional classification tasks, the autoregressive nature of LLM makes it a multi-step, multi-class, multi-label classification task. The overall objective is much more complex than a simple classification task.
- Multi-step:
- predict token 1 based on token 0
- predict token 2 based on tokens 0, 1
- predict token 3 based on tokens 0, 1, 2
- ...
- Multi-class:
- There are around
100,000classes (tokens)
- Multi-label:
- For the same input, the output is not unique, for example:
- Input:
“I love” - Output:
Dog,Cat,LLM, ...
Therefore, in essence, it’s not really predicting an absolutely correct answer like in traditional classification tasks like Cat&Dog classification. It’s more like
scoring or ranking.In a given context, some words or tokens have a higher probability of occurring, though not necessarily a probability of 1, while others have a lower probability, but not zero.
When the context changes, the probability distribution shifts accordingly. However, no token is ever guaranteed to be the next one, nor is any token ever completely ruled out.
Therefore, the task that LLMs are trying to solve is a regression task.
Prev
10.5 What is QK-Norm?
Next
10.7 Dropout and DropPath
Loading...