Rao Blackwellization

Worked Example

Let X and Y be independent Bernoulli(p) random variables. We want to estimate the parameter target r = p(1-p).

1. Consider the naive estimator T = X(1-Y). The expectation is E[T] = E[X]E[1-Y] = p(1-p) = r, so it is an unbiased estimator.
2. The variance of this naive estimator (an indicator variable) is r - r².
3. Now consider the sufficient count statistic S = X+Y for parameter p.
4. We condition T on S. If S=0 (pair 00), T must be 0, so E[T|S=0] = 0. If S=2 (pair 11), T must be 0, so E[T|S=2] = 0.
5. If S=1, the outcome is either 01 or 10. By symmetry of the i.i.d Bernoulli distribution, each occurs with probability 1/2 given S=1. T is 1 for 10 and 0 for 01. Thus E[T|S=1] = 1/2(1) + 1/2(0) = 1/2.
6. The improved estimator is thus parameter-free: it only depends on the observed data S. Its variance over the full sampling law reduces to r/2 - r², a reduction of exactly r/2.

Experiment

You can interactively explore these estimators across the full sampling space using the Rao Blackwellization module.

1. Set the Bernoulli probability to 0.25. The target r = p(1-p) is 0.1875.
2. Set the Observed pair to 10. The naive estimator T evaluates to 1, while the sufficient count S evaluates to 1.
3. Notice the improved conditional estimator E[T|S] evaluates to exactly 0.5.
4. Observe the variance comparison across the full 4-outcome law. Both sampling expectations equal the target 0.1875. The naive variance is 0.15234375, while the improved variance drops to 0.05859375.
5. Change the Observed pair to 00. For this observation, S=0, so the conditional estimator evaluates to 0. However, the theoretical variances remain the same, because they evaluate the estimator performance across all possible samples, not just the currently observed pair.

Limitations

This exact example utilizes discrete Bernoulli pairs, leading to a closed-form parameter-free conditional estimator. In continuous or highly complex models, finding the closed form of E[T|S] is often analytically intractable, even if a sufficient statistic is known. Conditional Monte Carlo can exploit a computable conditional expectation; a numerical approximation adds its own error and needs a separate analysis. With finite second moments, the reduction is E[Var(T|S)]; it is strictly positive exactly when T is not a function of S up to probability-zero events. Finally, it is crucial to understand that while conditioning yields a parameter-free estimator (it can be computed from sample data alone), the true underlying parameter p remains unknown in practice.

References

The standard theory of sufficient statistics, Rao-Blackwellization, and sampling moments can be found in probability and mathematical statistics literature. Detailed derivations for Bernoulli count sufficiency, uniform conditional ordering, factorization theorems, control-variate centering and optimal coefficients are provided in the primary references below.

• Iowa STAT7400 notes: control-variate centering/optimal coefficient, antithetic covariance, and conditioning/Rao-Blackwellization sections - Luke Tierney
• MIT18.655 lecture 6 pages 3-4 and 8-9: Bernoulli count sufficiency, uniform conditional ordering and factorization theorem - Kempthorne