Direct Preference Relative Policy

Direct Preference Optimization (DPO) optimizes a language model policy by using a reference policy and a dataset of preferences. Instead of training a separate reward model, DPO formulates the reward directly as a function of the log-ratio of the policy probabilities relative to the reference probabilities.

Worked Example

Consider a setup where we have two possible responses to a fixed prompt. Let the reference probability for response 1 be q_ref = 0.5, and the current policy probability for response 1 be p = 0.5. Let beta = 1, and suppose we observe 4 comparisons where response 1 wins w = 3 times.

The empirical preference for response 1 is r = w / 4 = 0.75.
Calculate the logit of the reference probability: logit(0.5) = ln(0.5 / 0.5) = 0.
Calculate the logit of the policy probability: logit(0.5) = ln(0.5 / 0.5) = 0.
The DPO preference logit is d = beta * (logit(p) - logit(q_ref)) = 1 * (0 - 0) = 0.
The predicted preference is sigmoid(d) = sigmoid(0) = 0.5.
The average DPO loss is r * softplus(-d) + (1 - r) * softplus(d) = 0.75 * ln(1+1) + 0.25 * ln(1+1) = ln(2) ≈ 0.693147.
The policy-logit gradient is beta * (sigmoid(d) - r) = 1 * (0.5 - 0.75) = -0.25.
The optimal probability limit for this fixed r is sigmoid(logit(q_ref) + logit(r)/beta) = sigmoid(0 + logit(0.75)/1) = 0.75.

Experiment

You can interact with the Direct Preference Relative Policy module to observe how these values change. Follow these steps:

Set the Reference probability to 0.25.
Set the Policy probability to 0.5.
Set Beta to 1.
Set Response-one wins to 3.
Observe that the implicit reward difference becomes ln(3) ≈ 1.0986.
The predicted preference probability becomes 0.75, matching the empirical preference perfectly.
The policy-logit gradient becomes 0, indicating the loss is minimized.

Limitations

This module considers a simplified binary response scenario, which is not an actual language-model training or a reward-quality guarantee. When the empirical preference r is 0 or 1, the optimal policy limit is pushed to the boundaries (0 or 1). At these limits, there is no finite logit, and the minimum loss infimum approaches 0.

References

The following reference describes the Direct Preference Optimization framework:

Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. This paper introduces the DPO objective, showing how to optimize policies directly from preferences without an explicit reward model.

Direct Preference Relative Policy