Jensen Inequality

Move a two-point distribution and compare a convex function of its mean with its expected function value.

Jensen's Inequality is a fundamental result in probability theory that relates the value of a convex function applied to an expectation to the expected value of the convex function. For a random variable X and a convex function f, it states that:

f(E[X]) ≤ E[f(X)]

The difference E[f(X)] - f(E[X]) is non-negative and is known as the Jensen gap. The gap characterizes the extent to which the function deviates from linearity over the distribution of X.

Worked Example

Consider a two-point discrete probability distribution where X takes the value x = -2 with probability p = 0.25, and y = 2 with probability 1-p = 0.75. Let the convex function be f(t) = t².

First, calculate the expected value of X (Mean): E[X] = p(x) + (1-p)(y) = 0.25(-2) + 0.75(2) = -0.5 + 1.5 = 1.
Next, calculate the function of the expected value: f(E[X]) = f(1) = 1² = 1.
Then, compute the expected value of the function: E[f(X)] = p * f(x) + (1-p) * f(y) = 0.25 * (-2)² + 0.75 * (2)² = 0.25(4) + 0.75(4) = 1 + 3 = 4.
Finally, find the Jensen gap: E[f(X)] - f(E[X]) = 4 - 1 = 3.

We can analytically verify the gap for the square function, which expands to p(1-p)(x-y)². In this example: 0.25 * 0.75 * (-2 - 2)² = (3/16) * 16 = 3, matching our result.

Experiment

You can interact with a two-point distribution in the Jensen Inequality module.

Set Value x to -2.
Set Value y to 2.
Set Weight p to 0.25.
Set Convex function to Square.
Observe the Mean E[X] is 1, Function of Mean f(E[X]) is 1, Expected Function E[f(X)] is 4, and the Jensen Gap is 3.
Change Convex function to Absolute. Observe the Mean remains 1, f(E[X]) is 1, but Expected Function E[f(X)] is 2, giving a Jensen Gap of 1.
Change Value x to -1, Value y to 3, and Weight p to 0.5 with Square function. Observe Mean is 1, Function of Mean is 1, Expected Function is 5, giving a Jensen Gap of 4.
Set Weight p to 0 (Value x = -2, Value y = 2, Square). This creates a degenerate distribution entirely at y = 2. Observe the Jensen Gap becomes 0 because E[f(X)] and f(E[X]) are both 4.

Limitations and Assumptions

This visualization simplifies the concept by focusing strictly on a two-point discrete distribution, creating a simple convex chord. Jensen's Inequality applies to any probability distribution (discrete, continuous, or mixed) provided the expectations exist and are finite. The visual representation of the "gap" as a vertical line segment between the chord and the curve relies on interpreting the horizontal axis as the domain of the random variable and the vertical axis as the range of the function. For non-strictly convex functions (like the absolute value on an affine segment), equality (a gap of zero) can occur even when the distribution is not degenerate.

References

The properties of convex functions and expectations are commonly taught in probability courses. A thorough exposition of the discrete case can be found below.

Probability Course, Section 6.2.5: Jensen's Inequality (Hossein Pishro-Nik)

Jensen Inequality