3.5 Direct Preference Optimization (DPO)

3.5.1 Limitations of RLHF-PPO

• Information loss from the two-stage training process: RLHF involves first
• training a reward model using preference data, then
• applying PPO or another reinforcement learning algorithm to fine-tune the model.
If the reward model itself is biased—say, if the reward signals don’t align well with human preferences—then the subsequent reinforcement learning phase can further amplify this bias and degrade model performance.

• High resource demands of PPO algorithms:
• Reinforcement learning typically requires a process of exploration and exploitation, which can be unstable. To stabilize PPO training, various tricks are often needed, which increases complexity and reduces training efficiency.
• Additionally, PPO still relies on four components: Actor, Critic, Reward, and Reference models. These components are standard in classic reinforcement learning environments, where the Actor and Critic handle the policy and value estimation, respectively, and Reference ensures reward stability. However, at the LLM scale, all these modules need to be fine-tuned based on LLM SFT models. This significantly increases the number of trainable parameters and the overall computational cost, as well as potential cumulative errors introduced by the four-model setup.

3.5.2 DPO Derivation

3.5.3 DPO Code

3.5.4 Understand DPO via Advantages

🤔

You may be confused: if the final equation is the same, why bother?

This is because we are doing human preference alignment at the sentence level, which makes it a single-step MDP. But under multi-step MDP cases, you’ll see they are different.

For example, below is a simple grid world RL environment, where the agent receives a reward of -1 for each step and only gets a reward of 100 upon reaching the goal.

Now consider the two agent trajectories: path S (left) and path O (right). If we use the total reward as the metric to align with human preferences, then both trajectories have the same total reward:

So the preference model would treat them equally, but it's clear that path O should be preferred, because the agent is moving toward the goal, while in path S, the agent is moving away from it.

However, if we use advantage values as a preference metric, we can clearly predict that the cumulative advantage value in path O is larger, or in other words, the cumulative regret is smaller:

Actually, we could also use the action-value function , but advantage functions can be directly related to the optimal policy as we derived earlier.

3.5.5 Understand DPO via Contrast Learning

3.5.6 DPOP Algorithm

• If the current chosen answer’s sampling probability in the SFT reference model > the sampling probability in the current Policy model, minus a regularization factor (which means the current policy hasn’t been optimized well on the chosen answer yet, so we give it a penalty, which increases the loss);

• If the chosen answer’s sampling probability in the Policy model is higher, it indicates that the Policy model has already fitted this chosen answer well; so, at this point, we don’t give the additional penalty.

3.5.7 TDPO Algorithm

Simply put, since KL is an asymmetric distance function, forward and reverse under this case essentially mean whether the sampling probability is calculated based on SFT reference model or calculated based on the Policy Model, respectively.

3.5.8 Self-Reward

Experimental results show that both instruction following and reward modeling improve significantly through self-reward training. On the AlpacaEval 2.0 leaderboard, self-reward-based models show a clear win-rate boost.

• Compared to using IFT alone (i.e. the SFT baseline), adding EFT tasks into training (i.e. the self-rewarding method) does not affect the instruction-following ability (30.5% win rate vs. 30.9% win rate). This indicates that improving the model’s reward capability does not negatively impact other abilities.

• You can observe that with each iterative round, the performance steadily improves.