Law of Large Numbers

The sample average converges to the expected value as the number of trials increases.

Law of Large Numbers

Concept Overview

The Law of Large Numbers (LLN) is a fundamental theorem in probability that describes the result of performing the same experiment a large number of times. According to the law, the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer to the expected value as more trials are performed. This principle is why casinos always make money in the long run, and why insurance companies can reliably predict their payouts.

Mathematical Definition

Let X₁, X₂, …, X_n be an infinite sequence of independent and identically distributed (i.i.d.) random variables with expected value E(X_i) = μ and finite variance Var(X_i) = σ². The sample average is defined as:

X_n = (1/n) · Σ_i=1ⁿ X_i

The Law of Large Numbers states that:

X_n → μ as n → ∞

This convergence means that as the sample size n grows, the observed sample mean X_n gets arbitrarily close to the true population mean μ.

Key Concepts

Weak Law of Large Numbers (WLLN)

The Weak Law states that the sample mean converges in probability to the expected value. For any positive number ε (no matter how small), the probability that the absolute difference between the sample mean and the expected value is greater than ε approaches zero as n approaches infinity.

lim_n→∞ P(|X_n - μ| > ε) = 0

Strong Law of Large Numbers (SLLN)

The Strong Law states that the sample mean converges almost surely to the expected value. This means that the probability that the sequence of sample means converges to the expected value is exactly 1.

P(lim_n→∞ X_n = μ) = 1

Gambler's Fallacy

A common misunderstanding of the LLN is the Gambler's Fallacy—the belief that if something happens more frequently than normal during a given period, it will happen less frequently in the future to "balance out." The LLN does not imply that previous outcomes affect future ones; independent events remain independent. The convergence happens because the sheer volume of future trials drowns out the early fluctuations.

Historical Context

The concept was first formulated by the Italian mathematician Gerolamo Cardano in the 16th century, though without a rigorous proof. The first published rigorous mathematical proof of a form of the law (specifically for Bernoulli trials) was given by Jacob Bernoulli in his monumental work Ars Conjectandi in 1713. This specific case is often called Bernoulli's theorem. Later, Siméon Denis Poisson generalized it under the name "Law of Large Numbers" (La loi des grands nombres) in 1837.

Real-world Applications

Insurance and Actuarial Science: By pooling a large number of uncorrelated risks, insurance companies rely on the LLN to ensure that their average payout per policy closely matches the mathematically expected payout, allowing them to set profitable premiums.
Casinos and Gambling: Although individual players might win or lose big in the short term, casinos guarantee their long-term profit through the house edge. Because of the sheer number of games played, the LLN ensures their actual revenue per game converges to the expected value.
Monte Carlo Simulations: Computational algorithms use repeated random sampling to obtain numerical results. The LLN guarantees that as the number of random samples increases, the simulated estimate converges to the true value.
Machine Learning: When evaluating a model's performance on a test set, the LLN gives confidence that the empirical error (average loss over the test set) is a good estimate of the true generalization error, provided the test set is large enough.

Related Concepts

Central Limit Theorem: While the LLN states that the sample mean converges to the expected value, the CLT describes the distribution of the sample mean around that expected value.
Monte Carlo Simulation: A practical application that relies entirely on the Law of Large Numbers to compute complex integrals or probabilities.
Regression to the Mean: A distinct but related statistical phenomenon where an extreme event is likely to be followed by a less extreme one.

Law of Large Numbers