Simpson's Paradox

Compare within-group and pooled success rates, reveal aggregation reversals, and test a common population mix.

Concept Overview

Simpson's paradox describes an association that changes when observations are combined or separated into groups. In the strict reversal explored here, option A has a higher success rate than option B in each of two groups, yet B has the higher overall rate. The arithmetic is consistent: the options can have different proportions of observations in the two groups. This original fictional example isolates that mechanism without making a claim about any real treatment, population or experiment.

Mathematical Definition

For option t and group g, let s_t,g be successes and n_t,g be trials, with 0 ≤ s_t,g ≤ n_t,g and n_t,g > 0. The within-group rate r_t,g is a proportion. Multiplying by 100 expresses it as a percentage.

r_t,g = s_t,g / n_t,g

w_t = n_t,1 / (n_t,1 + n_t,2)

R_t = (s_t,1 + s_t,2) / (n_t,1 + n_t,2)

R_t = w_t·r_t,1 + (1−w_t)·r_t,2

A strict reversal requires r_A,1 > r_B,1 and r_A,2 > r_B,2, but R_A < R_B, or all inequalities in the opposite direction. The wider family of Simpson phenomena also includes associations that appear or disappear. This module labels only the strict two-group reversal; a tie or inconsistent group rankings is explicitly labeled “No strict reversal.”

Key Concepts

An example you can calculate by hand

In Group 1, A succeeds 9 times in 10 trials (90%), while B succeeds 72 times in 90 trials (80%). In Group 2, A succeeds 18 times in 90 trials (20%), while B succeeds once in 10 trials (10%). A leads by 10 percentage points in each group. Nevertheless, A's pooled rate is 27/100 = 27%, and B's is 73/100 = 73%.

Only 10% of A's trials belong to the high-rate Group 1, compared with 90% of B's. The actual mixtures are A: 0.1·90% + 0.9·20% = 27%; B: 0.9·80% + 0.1·10% = 73%. Averaging the two rates equally would silently replace the observed population with another one. The composition bars show these different trial weights separately from the success-rate bars.

Use the same weights to ask a different question

The common-mix slider applies a chosen Group 1 share w to both options. It computes a standardized descriptive rate, not additional successes or newly observed trials.

S_A(w) = w·r_A,1 + (1−w)·r_A,2

S_B(w) = w·r_B,1 + (1−w)·r_B,2

S_A(w) − S_B(w) = w·(r_A,1−r_B,1) + (1−w)·(r_A,2−r_B,2)

At w = 0.5, the example gives A = 55% and B = 45%. At w = 0.25, it gives 37.5% and 27.5%. The advantage stays at 10 percentage points because both group differences are exactly 10. More generally, if both group differences are positive, every common mix with 0 ≤ w ≤ 1 preserves a positive difference. Different weights for A and B remove that guarantee. The Balanced example has 50 trials in every cell, so the observed pooled rates already use common weights and no reversal occurs.

Undefined rates, ties and precision

A group with no trials has no observed success rate; 0/0 is undefined. All four group comparisons are therefore withheld until every group has positive trials. Successes must be whole counts between zero and the number of trials, and this learning tool limits each count to 10,000. Zero successes and all successes are valid. The screen rounds percentages to three decimal places, but the strict ranking compares integer cross-products s_A,g·n_B,g and s_B,g·n_A,g, so a rounded display does not turn an exact tie into a reversal.

Association does not decide causation

A pooled comparison describes the observed populations, and a standardized comparison describes a common population mix. Neither is automatically the right estimate of what an intervention would do. Adjusting for a relevant pre-existing confounder can help under appropriate causal assumptions; conditioning on a consequence of the option or a common effect can instead introduce bias. Deciding what to adjust for requires the data-generating context and assumptions, not a rule that the group-by-group result must always be preferred. These point estimates also do not establish statistical significance or measure sampling uncertainty.

Historical Context

Edward H. Simpson's 1951 paper studied the interpretation of interaction in contingency tables. Related aggregation problems had appeared earlier in work by Pearson and Yule; the phenomenon is also called the Yule–Simpson effect. Later causal analyses, including Judea Pearl's, explain why the same observed tables can require different interpretations depending on how the groups and options arise.

Real-world Applications

Experiment reporting: Compare variant performance by traffic source or device and check whether variant populations have different compositions before interpreting an aggregate conversion rate.
Population statistics: Age-specific rates and overall rates answer different descriptive questions when populations have different age distributions. Common weights make the target population explicit.
Model assessment: Aggregate performance can conceal differences across subgroups. Group sizes, evaluation targets and uncertainty should accompany the comparison.

Related Concepts

Conditional Probability Tree — conditional probabilities and joint event weights.
A/B Testing — sampling uncertainty and significance, which are separate from aggregation reversal.
Correlation & Covariance — association and the limits of causal interpretation.

References

E. H. Simpson (1951), The Interpretation of Interaction in Contingency Tables — the original paper.
Judea Pearl, Understanding Simpson's Paradox, UCLA Technical Report R-414 (2013) — primary analysis of reversals and causal interpretation.

Simpson's Paradox