Generalized Advantage Trace

Calculating the advantage trace

We manually calculate the advantage trace with Mixed rewards [1, 0, 2], Approximate values [1, 0.5, 0.25, 2], Discount γ = 0.9, Trace λ = 0.5, Horizon = 3, and Truncated ending.

1. Identify inputs: r = [1, 0, 2], V = [1, 0.5, 0.25, 2].
2. Calculate TD residuals (δ_t): δ_t = r_t + γ * V_t+1 - V_t.
   • δ₀ = 1 + 0.9 * 0.5 - 1 = 1 + 0.45 - 1 = 0.45
   • δ₁ = 0 + 0.9 * 0.25 - 0.5 = 0 + 0.225 - 0.5 = -0.275
   • δ₂ = 2 + 0.9 * 2 - 0.25 = 2 + 1.8 - 0.25 = 3.55
3. Calculate Generalized Advantage Estimate (A₀): A₀ = Σ_t=0..H-1 (γ*λ)^t * δ_t. Here γ*λ = 0.9 * 0.5 = 0.45.
   • A₀ = (0.45)⁰ * 0.45 + (0.45)¹ * (-0.275) + (0.45)² * 3.55
   • A₀ = 1 * 0.45 + 0.45 * (-0.275) + 0.2025 * 3.55
   • A₀ = 0.45 - 0.12375 + 0.718875 = 1.045125
4. Lambda-return target: V₀ + A₀ = 1 + 1.045125 = 2.045125.

Experimenting with advantage traces

Explore the relationship between TD residuals and advantage estimation in the Generalized Advantage Trace interactive module.

1. Vary Trace lambda: Set Trace lambda to 1. Notice how A₀ now equals 3.078, which is exactly the Discounted bootstrap return (4.078) minus the baseline V₀ (1). Then set it to 0 and observe that A₀ simply equals the first TD residual (0.45).
2. Change Horizon: Reduce the Horizon to 1. The trace truncates early, leaving only a single TD residual of 0.45, regardless of the Trace lambda setting.
3. Test True Terminal: With Horizon 3 and Trace lambda 0.5, switch Ending to "True terminal". The bootstrap value V_H is forced to 0. A₀ changes to 0.680625 and the Lambda-return target to 1.680625 because the final reward is no longer supplemented by the subsequent value approximation.

Assumptions and Limitations

This deterministic toy illustrates weighting and calculation mechanics, not measured stochastic bias/variance or a trained policy.

It uses predetermined arrays for rewards and values, isolating the mathematical relationships in a single trajectory. In a real reinforcement learning scenario, these values would be estimates from a neural network, updated continuously as new transitions are collected. Negative advantages are meaningful and properly handled by the calculations, indicating actions worse than the baseline. No reward beyond the selected horizon is silently used.

References

Primary sources for generalized advantage estimation.

• High-Dimensional Continuous Control Using Generalized Advantage Estimation (Schulman et al., 2015): Introduces GAE, detailing the exponentially-weighted average of n-step advantage estimators to reduce variance while maintaining tolerable bias.