Cramér–Rao Biased Variance Bound

Worked Example

Consider a binomial random variable K ~ Binomial(n, θ) where we want to estimate the true probability θ. We use the estimator (K + a) / (n + 2a) with a pseudocount a = 1 and sample size n = 4. Assume the true probability is θ = 0.25.

1. The expected value of our estimator is (nθ + a) / (n + 2a) = (4(0.25) + 1) / (4 + 2(1)) = (1 + 1) / 6 ≈ 0.3333333333333333.
2. The bias is the expected value minus the true θ: 0.3333333333333333 - 0.25 = 0.08333333333333333.
3. The variance of the estimator is nθ(1 - θ) / (n + 2a)² = 4(0.25)(0.75) / 6² = 0.75 / 36 = 0.020833333333333332.
4. The Mean Squared Error (MSE) is Variance + Bias² = 0.020833333333333332 + (0.08333333333333333)² = 0.027777777777777776.
5. The Sample Fisher information I is n / (θ(1 - θ)) = 4 / (0.25 * 0.75) = 21.333333333333332.
6. The unbiased CR bound is 1 / I = 1 / 21.333333333333332 = 0.046875. Notice the MSE (0.0278) is lower than this bound!
7. The derivative of the expectation with respect to θ is n / (n + 2a) = 4 / 6 = 0.6666666666666666.
8. The bias-adjusted variance bound is (dE/dθ)² / I = (4/6)² / 21.333333333333332 = 0.020833333333333332. The variance exactly matches this adjusted bound.

Experiment

You can observe the tradeoff between bias and variance interactively in the module.

1. Set the True probability to 0.25, Sample size to 4, and Pseudocount to 1. Observe the MSE falls below the Unbiased CR Bound.
2. Change the Pseudocount to 0. The bias becomes zero, and both the variance and the MSE exactly equal the Unbiased CR Bound.
3. Change the True probability to 0.5 and the Pseudocount to 1. Notice the bias is zero at this specific point, but the variance still follows the lower bias-adjusted bound rather than the unbiased bound because the derivative of the bias is not uniformly zero.

Limitations

This demonstration assumes θ is within the open interval (0, 1). At the boundaries θ = 0 and θ = 1, the regularity conditions required for the Cramér-Rao bound fail, and the Fisher information is undefined. Additionally, this analysis uses the frequentist MSE framework rather than substituting posterior variance, which would require a Bayesian interpretation of the credible interval rather than repeated sampling distributions.

References

For a rigorous derivation of the Cramér-Rao lower bound, including the formulation for biased estimators, see:

• Gallager, R. G. Chapter 4: Parameter Estimation (MIT). This text details the conditions for the bound and explains how a non-constant bias derivative scales the achievable variance.