Binary Channel Mutual Information

Follow a noisy binary channel and compare the information carried by biased and balanced inputs.

Binary Channel Mutual Information

In information theory, a noisy channel is typically modeled by the transition probabilities from an input alphabet X to an output alphabet Y. The Binary Symmetric Channel (BSC) is a common model where the input bit is flipped with probability q. The channel capacity is the supremum of the mutual information between the input and output over all possible input distributions.

Worked example: Information and Capacity

Consider a Binary Symmetric Channel where the input X is a Bernoulli random variable with P(X = 1) = p, and the noise Z is an independent Bernoulli random variable with P(Z = 1) = q. The output is Y = X ⊕ Z.

Let the input probability p = 0.5, and the flip probability q = 0.25. We want to find the output probability, the mutual information I(X;Y), and the channel capacity C.

Calculate the output probability P(Y = 1):
P(Y = 1) = P(X = 0) * P(Y = 1 | X = 0) + P(X = 1) * P(Y = 1 | X = 1)
Since P(Y = 1 | X = 0) = q and P(Y = 1 | X = 1) = 1 - q,
P(Y = 1) = (1 - p) * q + p * (1 - q) = 0.5 * 0.25 + 0.5 * 0.75 = 0.125 + 0.375 = 0.5.
Thus, r = 0.5.
Calculate the binary entropies. The binary entropy function is H2(x) = -x * log2(x) - (1-x) * log2(1-x).
H2(r) = H2(0.5) = -0.5 * log2(0.5) - 0.5 * log2(0.5) = 1 bit.
H2(q) = H2(0.25) ≈ -0.25 * (-2) - 0.75 * (-0.415037) ≈ 0.5 + 0.311278 = 0.811278 bits.
Calculate the mutual information:
I(X;Y) = H(Y) - H(Y|X) = H2(r) - H2(q) = 1 - 0.811278 = 0.188722 bits.
Calculate the channel capacity:
C = 1 - H2(q) = 1 - 0.811278 = 0.188722 bits.

In this case, since the input distribution is uniformly balanced (p = 0.5), the mutual information achieves the channel capacity.

Experiment

You can explore the relationship between the input bias, noise level, and mutual information in the interactive experiment.

Try the following ordered instructions to see the effects on the results:

Select the Flip probability dropdown and set it to 0.5 (completely noisy channel). Notice that the mutual information drops to 0, regardless of the input probability. The channel destroys all dependence between the input and output.
Select the Input-one probability dropdown and set it to 0 (constant input). Notice that the mutual information is 0. Since the input has no uncertainty, no information is being transmitted.
First set Input-one probability to 0.25, then set Flip probability to 1. The mutual information becomes approximately 0.811278124459 bits and the channel capacity is 1 bit. The channel is deterministically flipping the bits rather than destroying the information.

Limitations

This visualization specifically models the Binary Symmetric Channel with binary inputs and outputs. It assumes that the noise is independent of the input and acts identically on both symbols (0 and 1). In real-world communication systems, channels may be asymmetric, have memory (where errors occur in bursts), or operate over continuous alphabets (like the Additive White Gaussian Noise channel).

References

This source establishes the foundation for calculating channel capacity.

Polyanskiy, Y., & Wu, Y. Information Theory: From Coding to Learning. Definition 2.1, zero/support conventions, Definition 3.1 and BSC example. https://people.lids.mit.edu/yp/homepage/data/itbook-export.pdf

Binary Channel Mutual Information