3.3 RLHF: Reinforcement Learning from Human Feedback

3.3 RLHF: Reinforcement Learning from Human Feedback

3.3.1 Introduction for Reinforcement Learning in NLP

Modeling NLP as an MDP

• Action : The token generated at time The action space consists of the entire vocabulary, so the action space size is .

• Policy : The probability of selecting action (i.e., generating a specific token) given the current state .

• State : The concatenation of the prompt and all previously generated tokens up to time . The initial state is simply the prompt itself.

• State Transition: Transitions are deterministic (not like “you may fall when you walk along the cliff” as in the Q-learning example). The next state is formed by concatenating the current state and the selected action:

• Reward and Value Function : Defined as in standard reinforcement learning: immediate rewards and the value function over states. After taking action in state , the agent receive reward and transitions to state .

NLP RL Optimization Objective

• represents the prompt

• represents the response

• is the reward assigned to response given prompt

• A policy derived from offline data (e.g., human demonstrations), or

• An initial policy from supervised fine-tuning (SFT).

3.3.2 RLHF Process

Research Background

• In many reinforcement learning tasks, the objectives are complex and reward functions are difficult to define, making it hard to convey human goals to agents.

• Inaccurate or biased reward functions can lead to agents exploiting them in unintended ways, resulting in behavior that does not align with human expectations, or even causes harm—a phenomenon known as reward hacking.

• Designing proper reward functions often requires substantial effort from domain experts.

• Existing approaches like inverse reinforcement learning or mimic learning are limited when dealing with complex behaviors, and directly using human feedback as rewards can be expensive.

Research Objective

• Capable of handling tasks where humans can recognize desired behavior, but may not be able to provide demonstrations.

• Allows non-expert users to guide the intelligent system.

• Can scale to large and complex problems.

• Efficient in terms of user feedback and cost.

Research Method

• Comparison Labels: Given two segments of agent behavior trajectories, and , humans are asked which one they prefer. Let denote the agent's observations/states, and denote the agent’s actions. The comparison label can be represented as follows, where 0.5 indicates equal preference:

• Preference Modeling: One goal of Reinforcement Learning from Human Feedback (RLHF) is to align the reward function with human preferences. This is done by training a reward model using human preference labels. The idea is to uncover a latent reward function that correlates with human preference, assuming that the likelihood of a human preferring a segment is related to the total (and possibly length-normalized) reward over that segment. Based on the Bradley-Terry model, the probability that a human prefers segment over is:

• Reward Learning: With the preference model trained and human preference data collected, we can then treat this as a binary classification problem to learn our reward function. The loss function is a standard cross-entropy loss:
If we denote positive (preferred) and negative (not preferred) samples as and , the loss can be rewritten as:

• Policy Learning: once the reward function is learned, any reinforcement learning algorithm can be used to learn a policy that maximizes this reward.

• Online Learning: This paper adopts an online RLHF method, where reward learning and policy learning occur alternately and continuously. The agent interacts with the environment to collect new trajectory data, which is then labeled by humans for next-round learning.

Research Conclusion

• Core Idea: The algorithm aligns the reward function with human preferences, guiding the agent's behavior to develop in a direction that meets human expectations. During training, the policy is simultaneously optimized to maximize the predicted reward. This enables the agent to learn behaviors aligned with human preferences even when the reward function is not explicitly defined.

• Advantages of the Feedback Approach:
• Easier to Provide: Compared to giving absolute numeric scores, it is easier for humans to compare behavior trajectories. This lowers the difficulty of providing feedback, making it easier for even non-expert users to participate in training intelligent agents.
• Richer Information: Behavior trajectories of intelligent agents contain sequential action information, which reflects the agent’s behavioral characteristics and trends more effectively than single states. Therefore, this is more helpful for learning human preferences and can provide more valuable information for aligning the reward function.
• Benefits of Online Feedback: Collecting feedback online allows the system to continuously acquire human preference data and update the reward model and policy accordingly. This avoids over-reliance on previously defined reward functions, which may be inaccurate or lead to undesirable behaviors. Continuous updating ensures the system adapts better to complex, dynamic environments.
One limitation is the high cost of collecting human-labeled preference data. Therefore, in LLM applications, many researchers are exploring automated preference labeling, such as in RLAIF, which leverages large language models to simulate human preferences.

3.3.3 RLHF + PPO in LLM Alignment

InstructGPT

• The Reward Model is trained on top of the SFT model, excluding the final unembedding layer. It takes in a prompt and response as input and outputs a scalar reward. Nowadays, this is typically implemented by adding a linear layer of shape [hidden_size, 1] to the base model to predict the reward.

• As shown in the diagram, for a single prompt, multiple candidate responses may be labeled, meaning the human annotators may rank more than two outputs. InstructGPT converts these rankings into pairwise preference data for training, which yield totally combinations.

• When InstructGPT applies PPO, it modifies the standard reinforcement learning objective. Originally, the goal of reinforcement learning is to maximize reward:
Where is the learned RL policy. Building on this, InstructGPT incorporates insights from prior work by adding a per-token KL divergence penalty between the RL policy and the SFT policy . This helps prevent the RL policy from drifting too far from the SFT model. This kind of constraint is common in Behavior-Regularized Offline RL:
Additionally, InstructGPT introduces pre-training data prior (PPO-ptx) to further reduce overfitting and maintain generalization on public NLP benchmarks. This results in the final objective:

The Four Key Models in RLHF

Reward Model

1. Initialization: The Reward Model is initialized from the SFT model, inheriting its language generation capabilities. A value head (usually a linear projection layer) is added on top of the model to produce scalar reward scores.

2. Reward Model Training Loss: The Reward Model is trained using pairwise preference data. For each sample, we are given:
• : the prompt
• : the chosen (preferred) response
• : the rejected (less preferred) response
We train the model to assign a higher reward score to the preferred response. The loss is defined as:
Where:
• is the sigmoid function.
• Equivalently, the loss can be rewritten as:

Note that this loss used in InstructGPT is different from the one used in Deep Reinforcement Learning from Human Preferences (RLHF for traditional RL), which has loss of and more granularity during reward estimation because its Reward Model estimates are functioning like a latent factor.

This is effectively a binary cross-entropy loss between the two responses.

3. Difference from Traditional RLHF Rewarding: In the LLM-based RLHF setup, we often assign a single scalar reward to an entire response , rather than providing token-level rewards. This is a practical design choice that simplifies the training process and aligns with how human preference data is usually collected. Besides, in traditional RLHF, the two trajectories may not be the same, but in LLM RLHF, the prompt is the same for both positive and negative responses.
More generally, we abstract out an aggregation function (AGG), denoted as:
These AGG operators in the list represent: sum, weighted sum, select the final one, and attention-based aggregation, respectively. In this case, the final-token value (AGG = -1) is used.

4. Two Perspectives on Prompt → Response as MDP
From here, we can extend to two understandings of the loss used in training Reward models. These help explain why aggregating to a single reward makes sense.
1. Perspective 1: Single-Step MDP. View the entire response as a single action:
There are no intermediate state transitions, and the trajectory is just one step. The reward function . In this case, AGG is trivial, so we can simply take the final token’s score (AGG = -1) as the reward. This is the most common and practical setup used in RLHF with LLMs.
2. Perspective 2: Multi-Step MDP. In this view, the generation process is modeled as a multi-step MDP:
• Each token is an action:
• The state includes the prompt and all previous tokens, forming an incremental sequence:
In traditional RL, we might assign a separate reward at each step . But in the LLM RLHF setting, we typically assign only a final reward based on the entire sequence. This can be seen like we apply a Transformer-based AGG operator over all intermediate representations.
Where denotes the intermediate reward signal or hidden representation at each token.
This formulation highlights that even in a multi-step MDP framework, the final reward is not isolated — it depends on the entire history, reflecting the global reward of the generated response.

Actor Model

• The SFT model has already been trained with human-labeled data.

• The Reward Model has been trained using preference comparisons.

• The Actor is now being fine-tuned using PPO (Proximal Policy Optimization), guided by reward signals from the Reward Model.

1. Initialization: The Actor is initialized from the SFT model, which has already been trained to generate helpful responses using human-labeled data.
This SFT model serves as the base for subsequent RL fine-tuning — it provides a reasonable starting point for further improving response quality using human preference signals

2. Training the Actor: During training, we input a prompt into the Actor model, which generates a full response. The prompt and response are concatenated, forming a sequence that is then used to compute loss and update the policy.
The Actor is trained using the PPO loss, which adjusts the policy based on the generalized advantage estimation (GAE). See Section 3.2.11 PPO Algorithm. The loss function is:
Where:
• : the current policy (i.e. the Actor model we're updating).
• : the previous policy used to keep the actor updated within a safety region.
• : the GAE, which tells us whether the action at state was better or worse than expected.
• : a clipping parameter that stabilizes training

3. Key Idea: The GAE is a signal (computed using the Critic model ) that guides how we should update the Actor's policy . It’s like feedback saying “this action was good” or “this action was bad.” The Actor loss function treats current polocy as the learnable variable, and optimizes it to increase the likelihood of good actions and reduce bad ones. Note that this loss is computed token by token, meaning each output token has its own advantage score and loss.

Critic Model

• The SFT model has already been trained.

• The Reward Model has also been trained and frozen.

• Now, the Critic model is brought in to help compute advantages, complementing the reward signal from the Reward Model.

1. The Critic Model can be initialized in multiple common ways:
• From the SFT model, just like the Actor and Reward Model. In this case, a value head (typically a linear projection layer) must be added on top to predict scalar values.
• Or, share some parameters with the Actor with an additional value head.
• Or, directly initialized from the trained Reward Model, since both models predict a scalar score for the prompt-response input — making them structurally compatible.
This allows the Critic to either start from the same base as the Actor or take advantage of the Reward Model’s learned value representation, depending on training setup or specific preferences.
While the Critic and Actor models originate from the same base, they are treated as independent networks during PPO training and do not share parameters. For more details, please refer to Actor-Critic.

2. Value Estimation via TD Target: The Critic Model is trained to reduce the TD-error (temporal difference error), which compares the current value estimate to a one-step bootstrapped return:
The TD-error formula:
This teaches the Critic to align its prediction with the observed reward plus the estimated value of the next state.

3. In Practice: GAE-based Return Target. In practice, especially when using PPO, we don’t compute TD-error directly. Instead, we also involve GAE. Once GAE is computed, we define the return as:
This is return is an expanded TD target using multiple steps. Then, the Critic loss becomes:
Again, although this loss seems to depend only on the advantage, both the GAE and value in the are computed in previous steps. Therefore, we are actually making the current critic estimates to be closer to the action-value function derived from the previous critic.

Reference Model

1. Initialization: The Reference Model is usually initialized from the SFT model, just like the Actor. However, during PPO training, the Reference Model is completely frozen — its parameters are not updated. It simply provides a fixed probability distribution to compute token-level KL divergence against the Actor’s output.

2. KL Penalty and Adjusted Reward: In PPO, the optimization objective incorporates a KL penalty to discourage the Actor policy from diverging too far from the Reference policy :
Here:
• is the scalar reward provided by the Reward Model,
• is the KL penalty coefficient,
The second term measures how much the Actor has diverged from the Reference.

3. Token-Level Reward Decomposition: This KL-regularized objective can be decomposed into token-level rewards:
Where:
• is the time step (token position),
• is the final token (end of sequence),
• Only the final token gets the Reward Model’s scalar reward
• All tokens get a KL penalty to keep them close to the reference behavior.
Alternatively, this can be written more compactly as:
Where is an indicator function that returns 1 only if the current state corresponds to the final token (i.e., the token where the response ends).

Online & Offline RLHF

1. Online RLHF: The core idea of Online RLHF is to let the model generate its own responses, and then we score the quality of those responses to guide model updates.
• The model actively generates answers.
• It learns from human feedback on its own outputs.

2. Offline RLHF: In contrast, Offline RLHF does not require the model to generate answers on its own. Instead, it learns from a fixed dataset of collected responses, with associated preference scores (i.e., good vs. bad samples).
• This method is based on pre-collected, preference-labeled examples.
• It's known as off-policy training, where the model only learns from the given data — not by interacting with the environment/prompts to generate data/responses.
Benefits:
• The training is more stable on such fixed datasets.
• Ideal case: you find a large set of preference data that matches the model’s ability (not too hard, not too easy) — this maximizes sample efficiency.
In contrast, in Online RLHF, all training samples are generated by the model itself, and preference labels are given by humans.

3.3.4 PPO Training Tricks

• Model-Level: Techniques that directly modify how the model behaves during learning, such as loss design and reward shaping.

• PPO-Level: Techniques related to how rewards are estimated and optimized using PPO.

Model-Level Tricks

1. Token-Level KL Penalty: To avoid catastrophic divergence from the pre-trained model, PPO training introduces a penalty term based on the KL divergence between the output distributions of the RL model and the original SFT model. This is applied per token during training. Formally, the modified reward at each token step is:
Where the KL divergence is calculated as:
This penalty discourages the RL model from straying too far from the SFT model, thereby stabilizing training and preserving previously learned knowledge. This has become a standard to do LLM PPO.

Note: In traditional online reinforcement learning (which does not rely on a reference policy like SFT), a common technique is to add an auxiliary penalty term: . This term is not part of the reward, but serves as a way to encourage the model to increase the probability of the selected action . Here's how it works:

• If is very small, the loss becomes large, strongly urging the model to increase the probability of .

• If is already high, the loss becomes small, gently reinforcing the choice.

2. Generalized Advantage Estimation (GAE): Recall that . In practice, usually set or close to 1 to make GAE equivalent to Monte Carlo estimation, which is:
The full reward is usually only available at the end of the sequence (e.g., based on user feedback or scoring), so bootstrapped TD estimates () are not helpful. Using full-sequence rewards aligns better with the token-level training setup.

3. Adding SFT Loss: Combine the PPO objective with the original SFT loss. This means the model is not only learning from RL signals but also continues to minimize the next-token prediction loss. This helps in two ways:
• Preserves the language modeling abilities from the SFT phase.
• Reinforces consistency with pre-trained behavior, especially useful when the RL reward is sparse or noisy.

PPO-level Tricks

4. PPO-ptx: Pretraining Loss during PPO (from InstructGPT). In InstructGPT, the PPO-ptx technique builds on the original PPO objective (which includes KL-penalized reward maximization), and adds an extra objective: training the policy on pretraining data again during PPO. This additional loss is referred to as the ptx loss.
This is done to prevent the model from forgetting general knowledge learned during the pretraining phase. In other words, it mitigates the risk that RLHF will cause the model to overfit to human preferences at the cost of general language modeling abilities. The modified objective becomes:
Where:
• The first term is the standard PPO reward with KL regularization.
• The second term is the additional log-likelihood on pretraining data, used to preserve general capabilities.
• controls the strength of the ptx loss.
This approach is designed to reduce Alignment Tax — RLHF aligning the model to human preferences may cause it to lose some general NLP capabilities and perform worse on some NLP benchmarks.

Alignment Tax & Model Size Sensitivity: The study by Anthropic, "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback", evaluates models ranging from 13M to 52B parameters across 7 different sizes. Their findings show that smaller models suffer more from alignment tax — their performance drops significantly after RLHF. In contrast, larger models (especially those in the 13B–52B range) retain or even improve downstream performance after PPO. This suggests that alignment tax is size-sensitive, and larger models are more robust to these trade-offs. Additionally, they empirically identified the optimal KL coefficient as .

5. KL Reward Scaling: The second term in the KL-based reward function includes a scaling factor . Based on practical training experience, the choice of is very important — it helps avoid scenarios where the policy diverges too far from the reference (which can cause overfitting or collapse).
To set appropriately, it is often tied to a target KL divergence value. Ideally, we want the actual KL between the current policy and the reference policy to stay around this target. However, due to randomness from different training settings and model initializations, the ideal target KL can vary significantly. We can first empirically determine the best target KL based on experiments, then dynamically determine the :
Where:
• is the relative deviation from the target KL (clipped to avoid excessive changes).
• is the current KL penalty weight.
• is a tuning coefficient; in practice, they used .

Reference: Fine-Tuning Language Models from Human Preferences

2. PTX Loss (Pretraining Loss): Adding back the pretraining loss during PPO training. Just like the KL penalty term, this technique introduces a weight coefficient . For example, InstructGPT sets , but based on practical experience, it's typically best to jointly tune the balance between the policy loss and the pretraining loss for your specific task.

3. Reward Normalization: In RLHF training, applying reward normalization is often very helpful for training stability. Unlike in traditional game environments where rewards are well-defined and bounded, in language models, reward signals are usually generated by a learned model, and can vary wildly in scale. Normalizing rewards prevents rare large values from dominating training, and helps smooth the learning process.

4. Advantage Normalization (Single-GPU and Distributed): In PPO training, advantage normalization is also a widely used trick to stabilize training. Since raw advantage values can vary greatly in scale, we normalize them across a batch: (Note this is technically standardization, but conventionally people just say advantage normalization)
Where:
• is the batch mean
• is the standard deviation
This helps ensure consistent gradient magnitudes, improving learning stability.
In distributed training (e.g., using DeepSpeed or DDP), it’s important to compute the mean and standard deviation across the global batch (all processes) — not just on local batches. This is referred to as Distributed Advantage Normalization. Neglecting this can lead to inconsistent updates between different GPUs, harming training convergence and stability.

9. Model Initialization: To ensure efficient PPO training, it's standard practice to initialize the Actor model using the fine-tuned SFT model, and initialize the Critic model using the reward model. This setup provides a strong starting point for both policy and value functions and is considered the default setup in most RLHF pipelines.

6. Adam Learning Rate:
• For the Actor model, the Adam learning rate is typically set to be about one-tenth of the SFT model’s learning rate. For example, in OpenRLHF:
• SFT model uses:
• Actor model uses:
• The Critic model’s learning rate may be set to around twice that of the SFT model. A commonly used default is:
• Critic model:

7. Value Function Loss Clipping: To stabilize value function updates, PPO often uses clipping when computing the value loss:
This technique was already covered in 3.3.3 RLHF + PPO in LLM Alignment.

Reward Exploitation and Generalization Issues

• Reward Hacking: This occurs when the model learns to exploit weaknesses in the reward model. The train reward increases, but the actual quality (as judged by humans) declines. As also discussed in 3.3.2 RLHF Process, this means the reward model has been “hacked” — the model finds a backdoor to maximize the score without actually improving the output.

• Poor Generalization: If the model performs well on the training data (as measured by both reward and human evaluation), but performs worse on the test set, this indicates poor generalization. This typically happens when the model memorizes training patterns and fails to adapt to unseen examples — a classic overfitting issue.

3.3.5 RLAIF: Reinforcement Learning from AI Feedback

• Generating preference labels for which candidate output is better.

• Ranking multiple responses according to quality or alignment criteria.

• Directly suggesting improvements to the generating model (though this can be more computationally expensive).

RLAIF in Practice

1. (Optional) Train or Fine-Tune a Feedback Model
• RLAIF often starts with a strong existing LLM serving as the “feedback model.”
• For specialized domains (e.g., medical or finance), you might fine-tune this model further so it can better judge domain-specific terminology and nuances.

2. Generate AI Feedback
• Present the feedback model with a context and two (or more) candidate responses.
• Prompt the model to choose which response is better (or to rank them). The feedback model outputs a preference distribution.
• Position bias is often reduced by swapping the order of responses and averaging the preference scores.
• Some implementations use a chain-of-thought (CoT) approach: the feedback model first explains its reasoning, and then outputs its preference token (e.g., “1” or “2”).

3. Train the Preference Model
• Use the AI-labeled data to train a model that maps (context, response) → reward. This preference model is typically lightweight and optimized for fast inference, making it suitable for use inside the reinforcement learning loop.
• Alternatively, the feedback/model’s preference can be used directly (bypassing training a separate preference/reward model), but that is typically more computationally expensive because it requires generation in the training loop.

4. Reinforcement Learning with AI Feedback
• Incorporate the preference/reward model into a reinforcement learning loop, where the LLM’s outputs receive a reward signal from the model.
• Over many steps, the LLM is optimized to produce responses that score highly with respect to the AI-provided feedback.

Advantages of RLAIF

• Scalability: By automating feedback, RLAIF reduces dependence on costly human annotations. This can be critical for training very large models or working with tasks where human evaluation is labor-intensive.

• Flexibility
• The “principles” for good responses (the “constitution”) live in the feedback model’s instructions or training data. Altering these instructions or data sources is often easier than assembling brand-new human annotation pipelines.
• When switching domains or tasks, you can simply fine-tune or reconfigure the AI feedback model.

• Potential for Improved Performance
• Studies comparing RLAIF and RLHF show that RLAIF can match, or sometimes exceed, RLHF in terms of output quality and alignment with desired behaviors.
• By using carefully crafted prompts and preference signals during training, the feedback model guides the base LLM toward generating responses that are more helpful, harmless, and aligned with specific goals—such as policy compliance or tone consistency. This consistent supervision during training leads to higher rates of safe and aligned dialogue generation in the resulting model.

Challenges and Considerations

• Alignment with Human Values
• Although the AI feedback model’s “constitution” is explicit, the underlying LLMs remain black boxes. Ensuring the AI’s values match human ones remains essential.
• Human oversight should remain in the loop, especially for tasks with high-stakes ethical or safety concerns.

• Evaluating AI Feedback Quality
• It can be difficult to judge whether the AI feedback is truly aligned with human preferences, especially if the feedback model itself contains biases.
• Robust evaluation protocols are needed to confirm that AI judgments consistently match human intent.

• Addressing AI Bias
• Like human annotators, AI models can exhibit biases from their training data.
• Ongoing work is needed to detect, mitigate, and correct biased AI feedback that might skew a model’s behavior.