Rényi Entropy and Effective Support

Rényi entropy is a family of functionals for quantifying the diversity, uncertainty, or randomness of a system. It generalizes the Shannon entropy, the Hartley entropy, the collision entropy, and the min-entropy.

Worked Example

Consider a discrete distribution with probabilities p = (1/2, 1/4, 1/8, 1/8). We can calculate its collision probability and its order-2 Rényi entropy.

1. Collision probability: sum(p_i^2)
= (1/2)^2 + (1/4)^2 + (1/8)^2 + (1/8)^2
= 1/4 + 1/16 + 1/64 + 1/64
= 11/32 = 0.34375

2. Order-2 Rényi entropy:
H_2(p) = -log2(sum(p_i^2))
       = -log2(11/32)
       ≈ 1.540568

Experiment

In the Rényi Entropy and Effective Support module, you can analyze different distributions and see how the entropy changes.

Select Skew from the Distribution control to match our worked example.
Change the Order to 2 and observe the calculated Rényi entropy and collision probability.
Try Order Infinity to see the minimum entropy based only on the maximum probability.

Limitations

This module considers only finite discrete distributions up to 4 elements. The continuous generalization (differential Rényi entropy) may diverge or become negative, and is not covered here.

References

The original paper introducing these measures is available here:

On Measures of Entropy and Information (Rényi, 1961)

Rényi Entropy and Effective Support