Neyman–Pearson Randomized Test

Spend a false-positive budget on the outcomes with the strongest likelihood ratios.

Neyman–Pearson Randomized Test

The Neyman–Pearson lemma provides the optimal way to test a simple null hypothesis against a simple alternative hypothesis. It asserts that the test with the highest power for a given size α is a likelihood-ratio test. For discrete distributions, achieving the exact size α often requires randomizing the decision at the boundary.

Worked Example

Consider a single observation with three possible outcomes: 0, 1, and 2. We have the following probability mass functions under the null hypothesis (H₀) and the alternative hypothesis (H₁):

H₀: P(0) = 1/2, P(1) = 1/3, P(2) = 1/6
H₁: P(0) = 1/6, P(1) = 1/3, P(2) = 1/2

The likelihood ratios (LR = H₁/H₀) are:

Outcome 2: (1/2) / (1/6) = 3
Outcome 1: (1/3) / (1/3) = 1
Outcome 0: (1/6) / (1/2) = 1/3

To maximize power for a size α = 4/12 (or 1/3), we select outcomes in descending order of their likelihood ratios.

Outcome 2 (LR = 3): Has H₀ mass 1/6 (or 2/12). Since this is less than 4/12, we set its rejection probability φ₂ = 1. Remaining budget: 4/12 - 2/12 = 2/12.
Outcome 1 (LR = 1): Has H₀ mass 1/3 (or 4/12). Our remaining budget is 2/12, which is half of 4/12. Thus, we randomize at this boundary: φ₁ = (2/12) / (4/12) = 1/2. Remaining budget: 0.
Outcome 0 (LR = 1/3): Has H₀ mass 1/2. Budget is 0, so φ₀ = 0.

The actual test size is P_H0(reject) = 1 × (2/12) + 1/2 × (4/12) + 0 × (6/12) = 4/12 = 1/3. The power is P_H1(reject) = 1 × (6/12) + 1/2 × (4/12) + 0 × (2/12) = 8/12 = 2/3.

Experiment

Follow these ordered instructions using the interactive controls to observe how the randomized decision rule is constructed:

Adjust the Alpha twelfths slider control to 4 (which equals an α of 4/12).
Observe the mathematical results in the Rejection Probabilities section: φ₂ becomes 1, φ₁ becomes 1/2, and φ₀ remains 0.
Select an Observed outcome using its slider control to view the specific rejection probability for that particular data point.

Try a full-page version at https://riano.app/module/neyman-pearson-randomized-test to observe how boundary cases like α = 0 or α = 1 completely constrain or relax the test.

Limitations

The decision rule implemented here relies on exact known probability mass functions, which is typically true only for a simple null and a simple alternative hypothesis. In practice, composite hypotheses require more advanced techniques. Furthermore, randomized tests, while mathematically optimal, are rarely used in real-world applied statistics because practitioners prefer deterministic decisions rather than flipping a coin when data falls on a boundary.

References

PennState Eberly College of Science STAT 415. Lesson 26: The Neyman-Pearson Lemma. https://online.stat.psu.edu/stat415/lesson/26/26.1

The reference linked above details the formal proof of the Neyman-Pearson lemma and walks through continuous and discrete examples demonstrating the optimality of likelihood-ratio based decisions.

Neyman–Pearson Randomized Test