Covariate Shift Importance Weighting

Compute exact two-group risks, density-ratio weights and ESS dispersion, including unsupported target groups.

Covariate Shift Importance Weighting

Concept Overview

In machine learning, covariate shift occurs when the distribution of inputs (covariates) changes between the training (source) environment and the testing (target) environment, while the conditional distribution of outputs given inputs remains fixed. Standard empirical risk minimization on the source data can yield biased models if certain regions of the input space are underrepresented.

Importance weighting corrects this bias by reweighting the source samples according to the density ratio between the target and source domains: w(x) = q(x) / p(x).

Worked Example

Consider a binary input G ∈ {0, 1}. We define the source distribution by P(G=1) = 0.5 (and thus P(G=0) = 0.5), and the target distribution by Q(G=1) = 0.8 (and Q(G=0) = 0.2).

Assume a fixed conditional loss profile A, where the expected 0-1 loss for group 0 is e₀ = 0.1 and for group 1 is e₁ = 0.9. We expect the risk on the target to be 0.74. Let's calculate the source risk and weights.

Source Risk: The unweighted risk under the source distribution is:
R_src = P(G=0) · e₀ + P(G=1) · e₁
R_src = 0.5 · 0.1 + 0.5 · 0.9 = 0.05 + 0.45 = 0.50
Oracle Target Risk: The true risk under the target distribution is:
R_tgt = Q(G=0) · e₀ + Q(G=1) · e₁
R_tgt = 0.2 · 0.1 + 0.8 · 0.9 = 0.02 + 0.72 = 0.74
Importance Weights: We compute the density ratios w_g = Q(G=g) / P(G=g):
w₀ = 0.2 / 0.5 = 0.4
w₁ = 0.8 / 0.5 = 1.6
Importance Weighted Source Risk: We reweight the source expected loss:
R_iw = P(G=0) · w₀ · e₀ + P(G=1) · w₁ · e₁
R_iw = 0.5 · 0.4 · 0.1 + 0.5 · 1.6 · 0.9 = 0.02 + 0.72 = 0.74

The importance weighted risk on the source data exactly recovers the true oracle target risk of 0.74.

Experiment

You can reproduce this calculation in the Covariate Shift Importance Weighting module:

Set Source group-one probability to 0.8.
Set Target group-one probability to 0.2.
Set Loss profile to A.
Set Source group-one probability to 0, keeping Target group-one probability at 0.2. The weighted risk is not identifiable and ESS is undefined. Set target probability to 0 as well: overlap holds again, the unused G=1 weight is zero, and risk is 0.100000.
Restore Source group-one probability to 0.8 and Target group-one probability to 0.2 for the numerical comparison below.

For these restored 0.8/0.2 settings, observe that the source risk is 0.74, while the oracle target risk drops to 0.26. The importance weights are w₀ = 4.0 and w₁ = 0.25, and the reweighted source risk matches the 0.26 target risk. Also observe the Effective Sample Size (ESS) fraction drops to 4/13 = 0.307692, an asymptotic weight-dispersion diagnostic rather than an actual finite sample count.

Limitations

This visualization assumes a simple binary input group and exact knowledge of the conditional expected loss, which in practice must be learned from finite samples. A critical limitation of importance weighting is the requirement of support overlap. If the target distribution contains mass where the source distribution has none (e.g., setting Source probability to 0 and Target to 0.8), the target risk becomes unidentifiable from the source alone. Furthermore, even when support holds, large weights can cause high variance in the empirical estimator, significantly reducing the Effective Sample Size.

References

Sugiyama, M., Krauledat, M., & Müller, K. R. (2007). Covariate Shift Adaptation by Importance Weighted Cross Validation. Read paper.

This paper derives importance weighting under covariate shift, where the input distribution changes but conditional outputs remain fixed. The module uses declared two-group conditional losses and exact expectations, not estimated density ratios or a finite-sample guarantee.

Covariate Shift Importance Weighting