Sufficient Statistics Factorization

Group binary samples by their success count and see which ordering information stops depending on the unknown probability.

Sufficient Statistics Factorization

In statistics, a statistic is sufficient for an underlying parameter if it contains all the information available in the sample about that parameter. The sufficiency principle states that if a sufficient statistic exists, any inference about the parameter should depend on the sample only through the value of the sufficient statistic. The Factorization Theorem provides a formal mathematical criterion for establishing sufficiency: write the joint mass as g_p(K(x))h(x), where h does not depend on p. Here g_p(k) = p^k(1-p)^4-k and h(x) = 1 on the sixteen binary tuples (and 0 outside that sample space).

Worked example: 4 Coin Flips

Consider four independent Bernoulli observations with an unknown probability p of success (1). Let the ordered observations be x_1, x_2, x_3, x_4 and the total success count be K = x_1 + x_2 + x_3 + x_4.

Suppose the probability of success is p = 0.25, and we observe the sequence 0110. Here, K = 2.

The likelihood of observing this specific ordered sequence is:

P(X_1=0, X_2=1, X_3=1, X_4=0) = (1 - p) · p · p · (1 - p) = p^2 (1 - p)^2

For p = 0.25, the ordered likelihood is 0.25^2 · 0.75^2 = 0.0625 · 0.5625 = 0.03515625.
Now, let's find the likelihood of the count K = 2. There are C(4, 2) = 6 possible sequences with exactly 2 successes: 1100, 1010, 1001, 0110, 0101, and 0011.
The count likelihood is therefore: P(K = 2) = 6 · 0.03515625 = 0.2109375
The conditional probability of observing the sequence 0110 given that the count is K = 2 is exactly: P(X = 0110 | K = 2) = 0.03515625 / 0.2109375 = 1/6

Notice that the parameter p cancels out completely. The conditional distribution of the data given the statistic K is uniform over all 6 sequences and depends only on K, not on p. This demonstrates that K is a sufficient statistic for p.

Experiment with Conditional Likelihoods

Open the Sufficient Statistics Factorization experiment to see how the factorization theorem applies to binary sequences.

Select a Bernoulli probability of 0.25 and an Ordered four-bit sample of 1010. You'll see the sequence belongs to the group K = 2. The conditional sequence probability is 0.166666666666667 with exact fraction 1/6.
Now, change the Bernoulli probability to 0.5. Notice that while the individual ordered likelihood and the count likelihood change significantly, the conditional sequence probability remains exactly 1/6 (with its displayed decimal approximation). Given K = 2, the identity of the specific ordering within the group K = 2 carries no information about p.
Change the sequence to 1000. It falls into the K = 1 bucket, which has 4 orderings. The conditional sequence probability is 0.25 (1/4), which is still independent of p.
Set the probability to 0 and the sequence to 1000. Since p = 0 implies K = 1 is impossible, the condition K = 1 has zero probability. In this case, the conditional law is unavailable, rather than dividing 0 by 0.

Clarifications and Limitations

It's important to distinguish the likelihood function from a probability distribution over the parameter space. The likelihood L(p; x) = P(X=x | p) is a function of p for a fixed observation x, but it need not normalize to 1 over p and is not a posterior distribution unless combined with a prior via Bayes' theorem.

The visualizer demonstrates sufficiency using only 4 samples for tractable enumeration. In practice, the factorization theorem applies to arbitrary sample sizes and continuous distributions (e.g., the sample mean is sufficient for the mean of a normal distribution with known variance). Moreover, while K is sufficient, observing only the first bit (e.g., X_1) generally discards information about p compared to observing K.

Sources

MIT 18.655: Mathematical Statistics (Spring 2016) Lecture 6, particularly pages 3-4 and 8-9, formalizes Bernoulli count sufficiency, uniform conditional ordering, and the factorization theorem.

Sufficient Statistics Factorization