✨ Awesome-Anything

Awesome-AI-Researchs

Awesome-GitHub-Repos

Awesome-AI-Examples

🖼️ Digital Image Processing

DIP I Image Demosaicing and Histogram Manipulation

DIP II Image Denoising

DIP III Edge Detection

DIP IV Digital Half-toning

DIP V Geometric Image Modification

🍃 LLM Components

1.1 Tokenization

1.2 Embedding

1.3 Attention

1.4 FFN, Residual Addition, and LN

1.5 Positional Encoding

1.6 Architecture & Decoding Policy

🌱 LLM Pre-training

2.1 Pre-training Definition

2.2 Pre-training Data

2.3 Pre-training Workflow

2.4 Continued Pre-training (CPT)

2.5 Pre-training Evaluation

☘️ LLM Post-Training

3.1 Supervised Fine-Tuning (SFT)

3.2 Reinforcement Learning

3.3 RLHF: Reinforcement Learning from Human Feedback

3.4 Optimizing PPO-Based Algorithms

3.5 Direct Preference Optimization (DPO)

🍀 LLM Popular Models

🪴 LLM Applications

5.1 Prompt Engineering

5.2 LLM-based Agents

5.3 Retrieval-Augmented Generation (RAG)

5.4 Enterprise-level RAG Practice (I)

5.5 Enterprise-level RAG Practice (II)

🌿 LLM Optimization

6.1 GPU Usage Analysis

6.2 Latency Analysis for Inference

6.3 FlashAttention

6.4 PagedAttention

6.5 Packing for Efficient Training

6.6 PEFT

🌾 LLM Compression

7.1 Quantization

7.2 Pruning

7.3 Knowledge Distillation

🌵 LLM Hands-on Practice

8.1 nanoGPT: Training a GPT from Scratch

8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example

8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis

8.4 OpenRLHF: Reward Model Training and Code Analysis

8.5 OpenRLHF: PPO Tutorial and Code Analysis

🌴 LLM Must-read Papers

9.1 Technical Reports Explained

9.2 MedLLM

9.3 o1 Techinical Paper

9.4 Prompting

9.5 LLM-as-Judge

🌳 LLM Q&A

10.1 Why GRPO training loss starts at 0

10.2 VLM-R1 Reproduction Results

10.3 KL(P||Q) or KL(Q||P)?

10.4 Left Padding or Right Padding?

10.5 What is QK-Norm?

10.6 Why LLM training rarely adopts Dropout?

10.7 Dropout and DropPath

🐝 VLM Image Encoders

11.0 Introduction

11.1 Vision Transformer

📝 MISC.

🤔 Large Language Models Pass the Turing Test.

OpenAI ChatGPT Progression

⏳ Claude Sonnet 4 Test with Sandtris

🔍 What DeepSeek-R1's Surge Teaches Us About SEO

💰 CivitAI’s Payment Issue

🎓 arXivSub, an Academia Assistant

3.4 Optimizing PPO-Based Algorithms

📌

PPO, as an online reinforcement learning algorithm, requires large-scale interactions with the environment (such as questions and answers) for sampling and policy updates. During RL training stage, four models are needed: Actor, Critic, Reward, and Reference models.

Among these models:

Actor (policy): require training

Critic (value function): require training

Reward model: only inference

Reference model: only inference

Training with these four models demands huge inference resources. As an on-policy reinforcement learning method, training PPO is not very stable. Some works have tried to improve RLHF-PPO algorithms for large language models.

Note: This section will not cover specific PPO implementation tricks

3.4.1 GRPO

Group Relative Policy Optimization

Paper: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Rethinking PPO Optimization

⭐

PPO refresher: In order to calculate Actor_loss, we need to use three models — Reference, Reward, and Value models. Among them, the Reference model participates in the KL reward calculation, and the Reward and Value models are involved in the GAE calculation. Here are the key equations, and you can also find them in Section 3.3.3 - Actor Model and Critic Model, and Section 3.2.11 PPO Algorithm for GAE calculation:

Actor (policy‑clip) loss:

Critic (value) loss:

Generalized Advantage Estimation (GAE):

🤔

Rethinking on PPO Optimization

If we add the Actor model, the PPO algorithm requires four models in total. In traditional reinforcement learning environments, the Actor, Critic, and Reference are all simple neural networks, and the Reward is a human-designed reward from the environment (which can be understood as rule-based reward). Thus, there is no large-scale resource requirement.

However, in the LLM setting, all models are either fine-tuned or initialized from LLM (SFT), meaning even if only Actor and Critic need parameter updates during PPO training, the inference and training of all four models still demand huge computation resources, and also has more accumulated errors across the models.

Let’s consider which of the 4 models could be removed:

First, the Actor model is essential. It represents the policy itself.

Next, it is possible to remove the reference model, then the training objective would focus purely on reward maximization. The reference model is typically kept to prevent the policy from diverging too far from the reference.

Then, for the Reward model, if we replace it with a rule-based reward, there would be no need to train a separate Reward model. However, whether rule-based or model-based, reward can’t be removed from the advantage calculation.

Finally, for the Critic (or Value model). Is it possible to replace or remove it?

Policy Gradient algorithm, Actor-Critic algorithm, and PPO algorithms all use value-based rewards to guide policy optimization. Typically, the Advantage function is used, and it’s originally defined as . It measures that at the current state , how much better or worse the current action is compared to the average action. Next, we will explore how GRPO removes the Critic model.

Advantage Function Estimation in GRPO

GRPO samples multiple outputs for the same question using , then the Reward model assigns a reward value to each of these outputs, and finally produces an estimate of the advantage function.

Outcome Reward Model (ORM): Using traditional ORM for reinforcement learning. The model generates a set of rewards . Then the advantage function is estimated by the following formula:

In this formula, it normalizes the reward within a sampled group and treats it as the advantage value, and assigns this normalized reward to each output tokens. Simply speaking, the reward for the current output is compared to the average reward of all outputs, and the difference between them is the advantage (the value can be positive or negative).

⭐

Example of ORM: Consider a simple math prompt: 10 × 10 = ?

We run the policy model 4 times, and get .

Reward model gives , where .

The correct answer 100 gets:

While other answers get:

Then, all tokens of 100 inherit 1.74, and every token of 95, 96, 124 inherits ‑0.58.

🤔

Why this method is valid?

Recall that in reinforcement learning, the value function is defined as the expected return (discounted future rewards):

Similarly, the action-value function is:

Under the setting of Outcome Reward RL:

The state is the prompt

The action is the output

And there is only a one-step reward

Thus, the value function simplifies to:

And

Therefore, if the number of samples is large enough, it is reasonable to estimate using , and estimate using . Then, the advantage estimation becomes . Dividing by is for normalization. However, the advantage obtained is for the entire output . GRPO uses (broadcasts) this advantage to every token of the output. In other words, for each token in output , the update direction is the same.

Process Reward Model (PRM): PRM estimates a sequence of rewards for each output token.

For output sequence , the reward is:

where is the length of .

And the reward for the all sequences can be denoted as :

Then, the advantage estimate for sequence and token position (step) is computed as:

⭐

Example of PRM: Consider a simple prompt: Explain 3 × 3 = 9 step‑by‑step

PRM grades each line of a reasoning chain:

step (token)	“3 × 3 =”	“(3 + 3 + 3)”	“= 9”	end
reward	0.1	0.2	0.5	1.0

After normalization, the final token reward is highest, and earlier tokens see the sum of their own and future rewards.

🤔

Explanation: For PRM, GRPO can’t be derived strictly following the value function definition. Instead, it is better to understand it through its formula. Here’s a simplified explanation:

For all sampled output steps where , rewards values are obtained based on PRM. Then, these rewards are used to calculate the advantage of each step.

However, this advantage calculation is performed individually for each step. In reality, the final output (or response) is a sequence of multiple steps. Therefore, the advantage of the current step should also account for the expected future outcomes. Specifically, the advantage value of the current step should be the sum of its immediate advantage and the advantages of all subsequent steps.

GRPO Loss Analysis

PPO loss: Formulas were introduced in Section 3.3.3 RLHF + PPO - Actor Model. Here is for a brief review. For every token in an output sequence that sampled from the old policy

Where is the advantage of token and is:

Adding a forward KL penalty using a fixed reference policy gives:

The extra term is the first‑order expansion (explain later) of around

GRPO loss keeps the clip trick but replaces the advantage and the KL divergence:

where

differs from PPO in two ways:

Group‑normalized advantage: (ORM or PRM) is computed as explained in the previous section.

Bias‑corrected KL estimation: for each token

which is a second‑order, non‑negative, unbiased approximation of the KL divergence, and it has lower variance.

🤔

Why GRPO’s KL divergence is called second‑order? Is it better? Why it is unbiased?

First‑order vs. second‑order:

Let , with is a very small number that approximate to 0.

Logarithm series

Expansion of the PPO’s KL divergence :

the leading term is first‑order in :

Expansion of the GRPO’s KL divergence

Define

Substitute and use the log‐series above:

Here the first nonzero term is , so GRPO’s KL divergence formula is second‑order in .

The GRPO term therefore measures the squared deviation rather than the linear deviation.

Why second‑order is better

Lower variance. is always greater than or equal to 0 and grows gently. On the other hand, is unbounded. Note that the symbol is natural logarithm, and the second-order function plot is:

Unbiasedness. Define a single sample random variable :

where is drawn from and

By definition, is unbiased, i.e., GRPO’s KL estimation is unbiased.

GRPO Code Analysis

GRPO’s code is from trl library: grpo_trainer.py.

Three key functions in GRPOTrainer: Below are simple code fragments that reflect the real code logic while remove auxiliary part (error checking, accelerator setup, etc.). Some code fragments are reordered with renamed variables from the source code to make it easier to understand.

_get_per_token_logps: computes the log-probabilities of the last tail_len tokens in each input sequence.

compute_loss: calculates the GRPO token-wise loss.

_prepare_inputs: This includes the key inovation of GRPO, i.e., estimating advantages (using ORM), where each response sentence only has a single scaler as the reward score

Group‑normalization (step 5) is the whole point of GRPO: it produces without a Value Model.

<ins/>

3.4.2 ReMax

ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

Policy Gradient and PPO Recap

⭐

Policy Gradient: You can view section 3.2.9 REINFORCE for more details.

is a reward-based estimation value, which provides the update magnitude (size) and direction (positive/negative) for the policy gradient .

The formula for expectation over multiple samples is:

If calculated based on multiple sample trajectories, it becomes:

⭐

Generalized Policy Gradient (section 3.2.10 Actor-Critic Algorithm) estimates by:

Estimator	Expression	Notes
Total return		Total discounted return of the entire trajectory.
Return		Monte-Carlo return used by REINFORCE.
Baseline-adjusted		Subtracting a baseline to reduce variance.
Action-Value		Hige variance and not stable, rarely used.
Advantage		In practice, using GAE for advantege estimation.
TD residual		Low variance, but biased estimation.

⭐

PPO (section 3.2.11 PPO Algorithm) builds on the advantage form from GAE and adds a ratio clip.

Where

The key part of PPO is still a policy-gradient method guided by the advantage function. The key difference in the loss calculation is:

PPO multiplies by the probability ratio , and then clips it as to constrain the update step;

REINFORCE multiplies by .

ReMax for LLM Alignment

⭐

ReMax: The initial goal of ReMax is the same as GRPO: to remove the Critic Model.

GRPO estimates the returns using advantages without critic/value model

ReMax changes the return estimation method of PPO from advantages to baseline-adjusted variant (a Monte-Carlo method)

Where the REINFORCE represents the baseline-adjusted variant part; and the Max represents the argmax operater we used for choose the best action, i.e., .

We previously discussed the REINFORCE method in policy gradients, where the algorithm directly estimates the return for using Monte Carlo sampling:

Where:

is just the gradient

is the number of sampled sequences,

is the sequence length.

However, in LLM reinforcement learning, there is usually only one reward value given at the final token (or final action) , and this reward can represent for the entire sentence. When adapting this strategy to LLM RL, REINFORCE becomes:

Where:

is the prompt,

is the response,

is the token output at (or action),

The state is concatenation of and :

However, as we mentioned earlier, the REINFORCE method using Monte Carlo estimation, which has large variance. So, ReMax uses a baseline-adjusted version:

The baseline is calculated as:

It means, for each output token, we use the reward of the greedy-decoded response as the baseline value. Reducing the baseline value can greatly lower the training variance, and this point has also been mentioned in section 3.2.10 Actor-Critic Algorithm.

Greedy decoding means that at each decoding step , you simply pick the single token with the highest probability under your current policy. For example, imagine you’re translating Hello into French. At the first step the model’s distribution might be
Bonjour: 0.7
Salut: 0.2
Coucou: 0.1
greedy decoding picks Bonjour without any randomness.

Normally, during on-policy fine-tuning, LLMs often use top-p = 1.0 or top-k = -1 sampling to encourage exploration, where k=-1 means sampling from all not just top-k; the greedy-decoded sentence reward is used as the baseline, because the SFT model has already produced reasonably consistent outputs across multiple generations for the same prompt, and the model's responses won't vary too much.

With baseline-adjusted approach, ReMax is much more stable than REINFORCE.

<ins/>

3.4.3 REINFORCE Leave-One-Out (RLOO)

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

After analyzing ReMax, RLOO is easier to understand. As the name suggests, it is also a variant based on the REINFORCE algorithm. Specifically, it modifies the baseline calculation:

For each prompt , it samples responses . Then, for each response, its baseline is set as the average reward of all other responses (excluding itself). Thus, for all prompts , the policy gradient becomes:

<ins/>

3.4.4 REINFORCE++

REINFORCE++: An Efficient RLHF Algorithm with Robustness to Both Prompt and Reward Models

Motivation: Critic-free RLHF methods like RLOO and GRPO estimate advantages separately for each prompt.

Note that the REINFORCE++ considers ReMax/RLOO as estimating advantages; but in our abstraction, we consider all these algorithms are estimating returns: PPO/GRPO do that by using advantages and ReMax/RLOO by using Monte-Carlo method. In this section, we just follow the terms used by the REINFORCE++ paper. Just a heads up so you don’t get confused.

For "easy" prompts, this can cause the model to focus too much on just one high-reward response, which can hurt generalization. REINFORCE++ addresses this issue by computing a baseline that reflects the average reward across the entire mini-batch, rather than calculating it independently within each prompt.

REINFORCE++ Method:

Sampling: Start from an initialized policy, but sample one response per prompt.

Token-wise advantage: The advantage is defined as the reward minus a KL penalty, which is proportional to the sum of token-level KL penalty from the step to the end of the sequence :

Where the token-level KL penalty at time is:

Batch-level normalization:

Where and are the mean and standard deviation of all advantages in the mini-batch.

Policy update – Plug into the usual PPO-style clipped objective. Still, no critic network is needed.

<ins/>

3.4.5 DAPO

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

📌

The balance between exploration and exploitation has always been a core issue in traditional reinforcement learning, because:

Exploration encourages agents (policies) to explore the environment and action spaces more, hoping to find better states and behaviors.

Exploitation focuses on using the current good policies to continue generating trajectories for learning and usage.

Reward Design can be seen as the soul of reinforcement learning. A good task reward design often reduces the cost needed for policy learning. However, it typically requires expert experience and fine-tuning, which is why RLHF methods come to exist to learn a reward model from human preferences. Recently, represented by DeepSeek-RL (section 4.7.5 Deepseek-R1), LLM RL has started returning to a more traditional design of scaler rewards.

DAPO is built on DeepSeek's GRPO. It further improves exploration-exploitation balance and reward design. And it mainly includes four key technical improvements.

⭐

Recap of GRPO

Where:

Before talking about the four improvements, DAPO removes the KL regularization in GRPO. KL regularization is typically used in offline RL and in LLM RLHF to prevent the policy from drifting too far from the behavior policy or initial policy (e.g., SFT policy). However, during the training of long-CoT (Chain of Thought) reasoning models, the policy model often diverges greatly from the initial policy. So, KL regularization becomes less meaningful and necessary, and was removed to allow more exploration.

DAPO’s formula is:

subject to:

where:

Clip-Higher — Raise the Ceiling

📌

As the name suggests, it raises the upper clipping threshold for .

Why do this? In the original PPO clipping method . If the current advantage is positive, it means we want to increase the probability of generating the current token in the response. However, for tokens that originally had very low probabilities, like , after clipping, the maximum policy update is , almost no increase. This can:

Limits the exploration of low-probability tokens

Limits the exploration of new reasoning patterns during long-chain thinking processes.

Thus, DAPO increases the upper clipping threshold and set , .

Note that for “exploitation token” with , the probability can’t be greater than 1. So, the Clip-Higher method doesn’t affect these tokens.

Dynamic Sampling — Keep Gradients Alive

📌

During the training process, before each training step, prompts are resampled, and prompts with accuracy 1 or 0 are filtered out. The whole batch consists of prompts whose accuracy is between 0 and 1.

For example, In GRPO, if all outputs in a group have an accuracy of 1, then based on the advantage calculation formula, the advantage for that group would be 0. This causes the gradient to vanish.

As training progresses, the number of outputs with 100% accuracy would gradually increase. So, to prevent the gradient vanishing problem and improve training stability, DAPO applies dynamic sampling, filtering out prompts with all outputs’ accuracies of exactly 1’s or 0’s.

💡

What is ?

This is a set that represents the outputs which are equivalent to a reference answer .

is a judgment function that returns whether is equivalent to the reference answer .
Therefore, this set contains all the correct .

What does the size of this set |·| represent?

is the number of correct outputs under a given prompt.

Suppose each prompt has outputs (e.g., generated candidates via sampling), then:

If this value is 0, it means all outputs under this prompt are incorrect (accuracy = 0).
If this value is , it means all outputs under this prompt are correct (accuracy = 1).

Therefore, with , we basically are excluding the prompts that lead to average accuracy of 0 and 1 from training.

The proportion of samples with average accuracy of 1

The training progress before and after applying dynamic sampling on a baseline setting

Token-Level Policy Gradient Loss

DAPO calculates the average loss over all tokens across all outputs within a group.

📌

In GRPO, when calculating the loss:

It first averages the token losses within each output,

Then averages the losses across all outputs.

This approach causes an imbalance in how outputs contribute to the total loss. Tokens from longer responses contribute less to the overall loss. This imbalance can cause two problems:

High-quality long samples: Harder to learn the good reasoning patterns.

Low-quality long samples (containing repetitive, irrelevant content, and thus have negative advantages): Harder to learn how to avoid generating such bad responses.

DAPO switches from sample-level loss averaging to token-level loss averaging. Make sure longer responses contribute proportionally to the final gradient.

Final observations show:

Token-level loss training is more stable

It prevents entropy from growing too high (which would cause the policy to behave randomly); Clip-Higher prevents the entropy from being too low and solves the under-exploration problem.

Overlong Reward Shaping

DAPO applies a soft punishment to penalize overlong responses by adding below negative reward to the accuracy reward:

Where and .

Experiments show smoother training and less entropy spikes:

Final Result

IME 2024 scores of DAPO with the Qwen2.5-32B base model, outperforming the previous SoTA DeepSeekR1-Zero-Qwen-32B using 50% training steps:

<ins/>

3.4.6 VAPO

VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks

Value-model-based Augmented PPO (VAPO) is proposed to train reasoning models. Using the Qwen 32B base model, it achieves a SOTA score of 60.4 on the AIME 2024 dataset, surpassing Deepseek-R1-Zero-Qwen-32B (score 47) and ByteDance's previous DAPO (score 50).

VAPO training is stable and efficient, reaching SOTA within just 5000 steps. It mainly studies how to perform Long-CoT (Chain of Thought) reasoning using the Value-model-based framework. It identifies and solves three key challenges:

Value Model Bias

Heterogeneous Sequence Lengths

Sparsity of Reward Signal

📌

Background: Value-model-based vs. Value-model-free

Value-model-free methods like GRPO, DAPO, etc., show great effectiveness in LLM reinforcement learning training. These methods estimate advantages by averaging rewards across multiple sampled trajectories in a batch, eliminating the need for an explicit value function (value model). However, in complex tasks, these methods show instability due to the high variance.

VAPO believes that, if the Value Model is accurately trained, Value-model-based approaches can have better performance. (This idea is similar to training a Reward Model vs. defining a rule-based model. If the Reward Model is accurately trained, its upper bound should be higher than the rule-based model, because it can provide finer-grained rewards).

A Value Model can precisely estimate the influence of each Action and the future accumulated Return, and has:

More accurate Credit Assignment
Fine-grained policy optimization

This is crucial for complex reasoning tasks, where even slight logical errors can lead to catastrophic failures.

Monte Carlo (MC) estimation (used by GRPO, DAPO, etc.) usually comes with high variance, while a Value Model can provide low-variance value estimates for each token, thus greatly improving training stability.

A well-trained Value Model leads to better generalization and better exploitation of different trajectories during reinforcement learning, raising the upper bound of RL performance.

🔎

Three Challenges of Training a Value Model for Long-CoT tasks

Value Model Bias over Long Sequences: For Long-CoT, it is very difficult to train a low-bias Value Model using bootstrapped methods. Specifically:

Monte Carlo estimation has low bias but high variance. Temporal Difference (TD) methods have higher bias but lower variance because they just "move one step”. We want to use GAE to balance bias and variance.
The VC-PPO paper mentioned that using a Reward Model (RM) for initial Value Model (VM) causes bias, because RM is trained to give rewards only at the <EOS> token. For the middle steps, RM gives relatively low rewards because the answer is not complete.

However, for VM, it should be able to estimate future expected returns for every token. So, if VM is initialized from RM, at the early training stage, the error will accumulate for GAE calculations.

Heterogeneous Sequence Lengths during Training: It's very challenging to handle short and long responses at the same time, as their impacts on bias-variance trade-off optimization are different.

For complex tasks, the length of the responses can be greatly different. Fixed hyperparameters (e.g., in GAE) are unsuitable for variable-length sequences.
For short responses, GAE tends to have high variance, while for long responses, GAE tends to have high bias (bootstrapping issues).

Example: Consider GAE with discount for simplicity and a fixed .

Short trajectory example (T−t=2): .

At , of the true “full‐return” signal still flows back. That means is almost pure MC, so it inherits almost all the sampling noise and high variance.

Long trajectory example (T−t=100):

Now only of the true return reaches step . is thus almost entirely bootstrapped from the next state value estimation , which is highly biased, like the TD residual method.

Sparsity of Reward Signal in Verifier-based Tasks: When using a verifier to provide binary (correct/incorrect) feedback only at the end of a generated reasoning chain, two factors make rewards extremely sparse as the length of CoT increases:

Many “zero-reward” steps

Intermediate tokens never receive any signal. Only the final token receives a reward (0 or 1).

A longer chain means more steps with zero feedback, so the agent sees almost no guidance during generation.

Fewer positive rewards overall

Longer reasoning chains are harder to get fully correct. One mistake anywhere means the whole chain is incorrect.

Therefore, balancing exploration (to sample diverse responses) and exploitation (to utilize correct responses) is very important.

Mitigating Value Model Bias over Long Sequences

VAPO proposes using Value-Pretraining and Decoupled-GAE to solve this problem, both introduced in the VC-PPO paper:

Value-Pretraining: It can be understood as a warm-up for the Value Model.

Start from a fixed policy. For example, continuously sample responses from a fixed policy . Then use the Monte Carlo return to train the Value Model directly, optimizing until the value loss and variance are sufficiently low.
Finally, save this Value Model to be used in the following RL training phases.

Decoupled-GAE: The main idea is to use different values when updating the Value and the Policy. Specifically:

When updating the Value, use . It means using accumulated rewards (MC method) to update the Value Model (assuming for simplicity). The formula is

This method solves the reward sparsity issue in Long-CoT tasks.

When updating the policy, GAE uses , a relatively lower λ value to accelerate the convergence of Policy Model.

📌

Why does a lower value accelerate the convergence of policy learning? By setting for the policy update, it has more bias but less variance of the advantage estimates. That lower-variance signal stabilizes gradients.

Managing Heterogeneous Sequence Lengths during Training

Mainly by using Length-Adaptive GAE and Token-level Policy Gradient Loss to address this problem.

Length-Adaptive GAE: For sequences longer than 100 steps, the TD-error coefficient associated with rewards would be , which is approximately zero. Therefore, fixed is not ideal for handling very long sequence outputs. VAPO uses length-adaptive λ:

Where is a hyperparameter, and is the sequence length.

Token-level Policy Gradient Loss: Mentioned in DAPO.

Handling Sparsity of Reward Signal in Verifier-based Tasks

To improve the exploration-exploitation balance in RL training, VAPO adopts three methods:

Clip-Higher: Mentioned in DAPO.

Positive Example LM Loss: Utilizing correctly answered samples during RL training

Compute the NLL (Negative Log-Likelihood) loss.

Add this loss to the PPO loss, enhancing learning for positive samples, through a weighting coefficient :

Group-Sampling: Instead of sampling one response per prompt, VAPO samples multiple responses per prompt. It introduces more contrastive signals into training, helping the policy model learn effectively.

<ins/>

3.4.7 TTRL

TTRL: Test-Time Reinforcement Learning

The main idea is to use Test-Time Scaling (TTS) to generate unlabeled data (e.g., test data without ground truth labels). Then, perform Majority Vote to automatically compute rewards for reinforcement learning optimization, which enables self-improvement of the model.

Note: this paper is not about a new RL method, just a way to augment data for RL

⚙

Detailed Procedure:

Based on the test dataset problems, sample N times with an LLM to get a group of responses (temperature set to 1.0 to enhance diversity, ).

Filter and Majority Vote over this group of answers (see majority voting in section 1.6.2 Decoding/Sampling Policy).

Choose the most frequent answer as the pseudo-label.

Then, calculate rewards:

If a response matches the pseudo-label, reward = 1.
If it does not match, reward = 0.

Use these rewards for LLM RL training based on the question-answer pairs. This paper mainly uses GRPO, and also includes PPO for comparison.

Throughout the whole process, no ground truth is used to compute reward. It leverages only the unlabeled data. (Note that in LLM, the ground truth is usually referred to as manually-labeled human preference in RLHF)

The pseudo-code of the majority voting reward function.

❓

Why is TTRL Effective?

Label Estimation: The key difference between TTRL and standard RL algorithms is that TTRL's rewards are inaccurate (because they are estimated through majority voting, similar to the model’s own confidence level). But it is still effective because:

Research has shown that reinforcement learning can tolerate a certain level of reward inaccuracy.

Rewards are often noisy and mainly serve as directional signals for exploration.

Previous research also found that more accurate reward models are not necessarily better "teachers". Therefore, self-estimated rewards from the policy model may provide better learning guidance.

Reward Design: According to TTRL’s reward design:

If the model’s response matches the majority vote, reward = 1;
Otherwise, reward = 0.

During early training (initially starting RL from the base model in this paper), even if the majority voting pseudo-labels are not very accurate, the reward signal still maintains a certain level of accuracy.

For example, we draw 8 predictions from the model [1, 1, 2, 2, 2, 4, 5, 6]

Majority-vote pseudo-label: the most common value in that list is 2, so we set . But the true label is 3.
Compute per-sample rewards

Estimated rewards: for each sample , give a reward of if , else 0.

True rewards: for each sample, give a reward of if (the real label), else 0. Since none of the eight draws is equal to 3,

Reward‐hit rate:

🤔

When Does TTRL Fail?

Reinforcement Learning Parameter Settings

Low-temperature coefficient: A lower temperature setting in the policy can reduce exploration.
Insufficient training for high-difficulty Questions: The model may not be able to handle them without well-training.

Lack of Task-Relevant Prior Knowledge or Model Is Too Small. TTRL is based on the model’s internal rewards (giving high rewards to answers that are more confident and appear more frequently). If the model’s capabilities are weak (or the task is too hard for the model), the reward signals will deviate from the task goal, and training will not be effective. Previous experimental results showed Qwen2.5-Math-1.5B and LLaMA-3.1-8B-Instruct performed worse under TTRL than Qwen2.5-Math-7B.

For example: Training an agent by traditional RL to find the treasure in a maze. External rewards are provided (like scoring when finding the treasure). But under TTRL, the agent only trusts itself, and rewards itself based on its own majority voting at the start. If it initially knows nothing about finding treasures, it may reward itself just for random wandering, without moving toward real goals.

<ins/>

3.4.8 Summary

Starting from REINFORCE, from top to bottom, you will see the tricks used in LLM RL training become more and more.

3.3 RLHF: Reinforcement Learning from Human Feedback

3.5 Direct Preference Optimization (DPO)

Last update: 2025-05-10

🧬 AI Learning Blog 🧬

Anticipated updates:

Must-read papers, and Hands-on practice in general
Digital Image Processing/Mathematics
Vision Language Model