Exact Permutation Test

Worked Example

Consider four units with outcomes [0, 1, 2, 3]. Suppose exactly two units are assigned to treatment. The possible treatment assignments are combinations of 2 from 4, which is exactly 6 assignments.

Under the sharp null hypothesis, the observed outcomes for each unit would have been the same regardless of treatment assignment. Let's calculate the test statistic T as the difference in means: T = mean(Y_T) - mean(Y_C).

1. Assignment (0, 1): Treated [0, 1], Control [2, 3]. Mean diff = 0.5 - 2.5 = -2.
2. Assignment (0, 2): Treated [0, 2], Control [1, 3]. Mean diff = 1.0 - 2.0 = -1.
3. Assignment (0, 3): Treated [0, 3], Control [1, 2]. Mean diff = 1.5 - 1.5 = 0.
4. Assignment (1, 2): Treated [1, 2], Control [0, 3]. Mean diff = 1.5 - 1.5 = 0.
5. Assignment (1, 3): Treated [1, 3], Control [0, 2]. Mean diff = 2.0 - 1.0 = 1.
6. Assignment (2, 3): Treated [2, 3], Control [0, 1]. Mean diff = 2.5 - 0.5 = 2.

If the observed treated pair is (2, 3), the observed test statistic is 2. The exact two-sided p-value is the proportion of permutations where the absolute test statistic is greater than or equal to the absolute observed test statistic. The absolute values are: 2, 1, 0, 0, 1, 2. There are 2 out of 6 permutations where the absolute difference is >= 2. Thus, the two-sided p-value is exactly 2/6 = 1/3 (0.333333).

Experiment

You can interact with the Exact Permutation Test module to see how the null distribution and p-value change based on the selected dataset and alternative hypothesis.

1. Set Dataset to Spread, Treated pair to 23 and Alternative to Two-sided. Read observed difference 2 and p = 1/3 from two extreme assignments.
2. Set Treated pair to 02. The observed difference is -1 and four assignments are extreme, giving p = 2/3.
3. Restore pair 23 and choose Alternative Greater. Only the +2 assignment is extreme, so p = 1/6.
4. Select Dataset Ties and Two-sided. Pair 23 has difference 1 and p = 1/3; the four zero statistics still represent four distinct labelled assignments.
5. Select Dataset Constant. All six statistics are zero, all are extreme and p = 1. There is no data-dependent deduplication.

Limitations

The reference distribution assumes exactly two of four labelled units were assigned uniformly to treatment under a sharp no-effect null. It is not a generic causal conclusion for observational groups. The observed assignment and every tie are included; full enumeration needs no Monte Carlo +1 correction.

• Computationally intensive: For large sample sizes, enumerating all possible permutations becomes computationally infeasible. In such cases, one must resort to Monte Carlo sampling (resampling) or asymptotic approximations.
• Sharp null only: The standard permutation test specifically tests the strong "sharp" null hypothesis that the treatment has absolutely no effect on any unit. It may not be robust if one is only testing the weak null hypothesis of equal means.

References

The official SciPy reference explains exact assignment enumeration and distinguishes full enumeration from randomized approximations. Riano independently constructs the four-unit examples and explicitly uses the stated absolute-statistic rule for its two-sided test.

• SciPy Documentation: scipy.stats.permutation_test