Gaussian Information Bottleneck Transition
A noisy representation starts preserving useful information only after relevance becomes valuable enough to justify the compression cost.
By Riano · Published · Updated · Report a correction
Gaussian Information Bottleneck Transition
The Gaussian Information Bottleneck defines a fundamental tradeoff for representations. A noisy encoder representation starts preserving useful information only after relevance becomes valuable enough to justify the compression cost.
Worked Example
Let's trace a concrete case of computing the optimal encoder signal fraction. We evaluate the parameters: Relevance correlation (ρ) = 0.8, Encoder signal fraction (s) = 0.5, and Relevance weight (β) = 2.
- Calculate input information I(X;T):
I(X;T) = -0.5 ln(1 - s) = -0.5 ln(1 - 0.5) = -0.5 ln(0.5) ≈ 0.34657359027997264 nats. - Calculate relevance information I(Y;T):
I(Y;T) = -0.5 ln(1 - ρ2s) = -0.5 ln(1 - 0.82 · 0.5) = -0.5 ln(1 - 0.64 · 0.5) = -0.5 ln(1 - 0.32) = -0.5 ln(0.68) ≈ 0.1928312404059923 nats. - Calculate bottleneck objective Loss (L):
L = I(X;T) - β I(Y;T) = 0.34657359027997264 - 2 · 0.1928312404059923 ≈ -0.03908889053201198. - Calculate optimal signal fraction (s0):
First, find the activation threshold βthreshold = 1 / ρ2= 1 / 0.64 = 1.5625.
Since β = 2 > 1.5625, the bottleneck is "active" (s0 > 0).
s0 = (βρ2 - 1) / (ρ2(β - 1)) = (2 · 0.64 - 1) / (0.64(2 - 1)) = (1.28 - 1) / 0.64 = 0.28 / 0.64 = 0.4375.
Experiment
You can interact with the analytical Gaussian bottleneck tradeoff using the interactive module:Gaussian Information Bottleneck Transition.
- Set Relevance correlation to 0.5.
- Observe the Activation relevance weight threshold changes to exactly 4.
- Set Relevance weight to 2 (which is ≤ 4). Notice that theOptimal signal fraction is exactly 0. The bottleneck refuses to allocate any bit rate because the relevance return does not justify the compression cost.
- Increase Relevance weight to 8. The representation now becomes active, finding a non-zero optimal signal fraction.
Limitations
This analytic Gaussian bottleneck is not a trained neural encoder or proof that all deep bottlenecks obey the exact same frontier. It assumes joint standard Gaussian variables with a linear Gaussian encoder. In cases where Relevance correlation (ρ) is 0, the variables are entirely independent, leading to an infinite threshold with no finite activation point.
References
The continuous phase transition behavior of the Information Bottleneck method is analyzed in detail in primary literature.
- Information Bottleneck for Gaussian Variables (Chechik et al., JMLR 2005) - Explains the explicit analytical solution and phase transition behavior for Gaussian variables.
Experience it interactively
Adjust parameters, observe in real time, and build deep intuition with Riano’s interactive Gaussian Information Bottleneck Transition module.
Try Gaussian Information Bottleneck Transition on Riano →