Prioritized Replay Importance Weights

Experience replay allows reinforcement learning agents to learn from past transitions. Instead of sampling uniformly, prioritized experience replay samples transitions with a higher magnitude of Temporal Difference (TD) error more frequently. To avoid bias, importance sampling weights are used to correct the expectation.

Worked Example

Consider four transitions with TD errors δ = [1, 2, 4, 8]. With priority exponent α = 1, correction exponent β = 1, and Raw normalization.

Calculate priority for each transition: p_i = |δ_i| + 1. So p = [2, 3, 5, 9].
Calculate sum of priorities: Σp = 2 + 3 + 5 + 9 = 19.
Calculate probabilities: q_i = p_i / Σp. So q ≈ [0.105, 0.158, 0.263, 0.474].
Calculate raw weights: w_i = (4 * q_i)^-1.
First sampling probability is approximately 0.105.
First raw weight: w₁ = (4 * (2/19))^-1 = 19/8 = 2.375.
Last sampling probability is approximately 0.474.
Last raw weight: w₄ = (4 * (9/19))^-1 = 19/36 ≈ 0.528.
Maximum weight is w₁ = 2.375.
Expected weighted error: Σ (q_i * w_i * δ_i) = (2/19)*(19/8)*1 + (3/19)*(19/12)*2 + (5/19)*(19/20)*4 + (9/19)*(19/36)*8 = 0.25 + 0.5 + 1 + 2 = 3.75.
Uniform target mean: (1 + 2 + 4 + 8) / 4 = 15 / 4 = 3.75.

Experiment

Use the interactive module to see how importance weights affect expected updates:Prioritized Replay Importance Weights

Set Errors to Increasing.
Set Priority exponent to 1.
Set Correction exponent to 1 and Normalization to Raw. Notice that the expected weighted error exactly matches the uniform target mean.
Change Normalization to Maximum. The expected weighted error is now scaled down compared to the uniform mean.
Change Correction exponent to 0. Notice that the expected weighted error diverges from the uniform target mean, revealing bias.

Limitations

This module is a simplified view of exactly four transitions with preset errors. It is not a full training agent. It demonstrates the expectation over a single sampling draw. In real applications, priorities are updated continuously, and the sum over the entire replay buffer must be maintained efficiently. Furthermore, using a correction exponent β < 1 intentionally introduces some bias to improve optimization stability, though it formally breaks the unbiased-variance guarantee.

References

The prioritized experience replay framework, including the priority definitions (with a small constant epsilon, approximated here by +1) and importance sampling weights, is presented in:
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized Experience Replay.https://arxiv.org/abs/1511.05952. This paper establishes the formulas for p_i, probabilities P(i), and importance sampling weights w_i.

Prioritized Replay Importance Weights