Doubly Robust Policy Value

Combine a reward model with importance-weighted residuals to estimate a target policy value.

Doubly Robust Policy Value

Policy evaluation in reinforcement learning and contextual bandits estimates the value of a target policy using logs collected from a different behavior policy. The doubly robust estimator combines two models: an inverse probability weighting (IPW) on the logging distribution and a direct reward regression (Q-hat).

Worked Example: Finite Log

Consider a log containing two contexts (0 and 1) and two actions (0 and 1). True rewards are deterministic: 0 and 1 for context 0, and 1 and 3 for context 1. Let the logging behavior be uniform, so each of the four (context, action) pairs occurs once with log-probability 0.5.

We want to estimate the value of a target policy that chooses action 1 with probability p = 0.25 (and action 0 with probability 0.75) in context 0, and chooses action 1 with probability 0.75 (and action 0 with probability 0.25) in context 1. Note: the target probability given as an input is for action 1 in context 0, and 1-p in context 1. If p = 0.25, the target policy takes action 1 with prob 0.25 in context 0, and 0.75 in context 1.

Assume the behavior estimate is "Correct" (uniform 0.5) and the reward estimate is a "Constant half" (0.5 everywhere).

1. Calculate Target Probabilities (pi):
   Context 0: pi(A=0)=0.75, pi(A=1)=0.25
   Context 1: pi(A=0)=0.25, pi(A=1)=0.75

2. Calculate Weights (pi / mu_hat):
   C0 A0: 0.75 / 0.5 = 1.5
   C0 A1: 0.25 / 0.5 = 0.5
   C1 A0: 0.25 / 0.5 = 0.5
   C1 A1: 0.75 / 0.5 = 1.5

3. Direct Estimate (mean_x sum_a pi * Q_hat):
   C0: 0.75*0.5 + 0.25*0.5 = 0.5
   C1: 0.25*0.5 + 0.75*0.5 = 0.5
   Mean: 0.5

4. IPW Estimate (mean_logged [ weight * reward ]):
   (1.5*0 + 0.5*1 + 0.5*1 + 1.5*3) / 4
   = (0 + 0.5 + 0.5 + 4.5) / 4
   = 5.5 / 4 = 1.375

5. Doubly Robust Estimate (mean_logged [ expected_Q + weight * (reward - Q_hat) ]):
   C0 A0: 0.5 + 1.5*(0 - 0.5) = -0.25
   C0 A1: 0.5 + 0.5*(1 - 0.5) =  0.75
   C1 A0: 0.5 + 0.5*(1 - 0.5) =  0.75
   C1 A1: 0.5 + 1.5*(3 - 0.5) =  4.25
   Mean: 5.5 / 4 = 1.375

The true target value is (3 - p)/2 = (3 - 0.25)/2 = 1.375. The Doubly Robust estimate matches exactly, providing unbiased evaluation when at least one model is correct.

Experiment with Estimators

You can test different nuisance models in the interactive Doubly Robust Policy Value module.

Set Target action-one probability to 0.25.
Set Behavior estimate to "Correct" and Reward estimate to "Constant half". Observe that the Doubly Robust value is exactly 1.375.
Change Behavior estimate to "Low action one". Because the reward estimate is wrong, the IPW estimator becomes biased (2.583). Since the reward model is also wrong, the DR estimator may be biased, but happens to remain correct in this particular deterministic fixture.
Change Reward estimate to "Zero". Now both nuisance models are incorrect, and the DR estimator yields a biased value of 2.583.

Limitations

The doubly robust estimator guarantees consistency if either the behavior model or the reward model is correct asymptotically. However, for a finite sample or specific logging probabilities, accidental equality can occur even if models are wrong. The provided fixture is deterministic and balanced; general random data do not guarantee exact equality or bounded estimates, particularly if importance weights become very large. This module performs no actual causal inference or policy training.

References

Dudík, M., Langford, J., & Li, L. (2011). Doubly Robust Policy Evaluation and Learning. ICML. Provides the foundational derivation and bias analysis of the doubly robust estimator for contextual bandits.

Doubly Robust Policy Value