Power Analysis

Power Analysis

Mathematical Definition

Statistical power is defined as the probability of correctly rejecting the null hypothesis (H₀) when the alternative hypothesis (H₁) is true. It is equal to 1 minus the probability of a Type II error (β).

The power of a test depends on four primary parameters: the significance level (α), the sample size (n), the population variance (σ²), and the true effect size (μ₁ - μ₀).

Key Concepts

Type II Error (β)

A Type II error (false negative) occurs when a statistical test fails to reject a false null hypothesis. In the context of the visualization, it is the area under the alternative distribution (H₁) that falls within the non-rejection region defined by the null distribution (H₀).

Effect Size (Cohen's d)

Effect size is a standardized measure of the magnitude of a phenomenon. For a difference in means, Cohen's d is commonly used: d = (μ₁ - μ₀) / σ. A larger effect size pushes the H₁ distribution further away from H₀, decreasing β and increasing power.

Sample Size (n)

Increasing the sample size reduces the standard error of the mean (σ / √n). This makes both the H₀ and H₁ distributions narrower and taller, reducing their overlap and significantly increasing statistical power.

Historical Context

The concept of statistical power was introduced by Jerzy Neyman and Egon Pearson in 1928 as part of their broader framework for hypothesis testing, which expanded upon Ronald Fisher's earlier work on significance testing. While Fisher focused exclusively on the null hypothesis and p-values, Neyman and Pearson introduced the alternative hypothesis, framing statistical testing as a decision-making process balancing Type I and Type II errors. Jacob Cohen later popularized power analysis in the behavioral sciences in the 1960s and 1970s, establishing standardized effect sizes (like Cohen's d) and providing accessible tables for researchers.

Real-world Applications

• Clinical Trials: Before starting a costly trial, medical researchers use power analysis to determine the minimum number of patients needed to reliably detect if a new drug is better than a placebo, ensuring ethical and efficient use of resources and human subjects.
• A/B Testing: Data scientists calculate power to determine how long an experiment must run (or how many users must be sampled) to detect a meaningful increase in conversion rates, avoiding premature conclusions based on underpowered tests.
• Psychological Research: Researchers use power analysis during study design to ensure they collect enough data to detect subtle behavioral effects, addressing the "replication crisis" driven by historically underpowered studies.

Related Concepts

• Hypothesis Testing — the fundamental framework within which power and errors are defined.
• Probability Distributions — specifically the Normal distribution, which forms the basis for the calculations and visualizations of Z-tests.