Noise-Contrastive Density Ratio

Theory

Suppose we have a data distribution p(x) and a known noise distributionq(x). We create a mixed dataset containing original data samples (labeled C=1) and noise samples (labeled C=0). For every 1 data sample, we generate knoise samples.

The prior probabilities are therefore:

• Data class prior: P(C=1) = 1/(k+1)
• Noise class prior: P(C=0) = k/(k+1)

The optimal Bayes classifier predicting the probability that a sample x came from the data class is given by the posterior probability:

D(x) = P(C=1|x) = [p(x)P(C=1)] / [p(x)P(C=1) + q(x)P(C=0)] = p(x) / (p(x) + k q(x))

If we have a perfect classifier that has learned D(x), we can rearrange this equation to recover the unknown data density p(x) in terms of the known quantities:

p(x) = k q(x) [D(x) / (1 - D(x))]

This is the fundamental population identity behind NCE. Note that if q(x) = 0(a support hole) and p(x) > 0, then D(x) = 1. The formula 1/0 becomes undefined, meaning we cannot recover p(x) in regions where the noise distribution has no support.

Worked Example

Let's verify the calculation manually. Assume state x=0, with uneven data and skewed noise.

1. Data probability at x=0: p(0) = 1/2
2. Noise probability at x=0: q(0) = 3/5
3. Noise samples per datum: k = 2
4. Optimal discriminator D(0) = (1/2) / (1/2 + 2(3/5)) = (1/2) / (1/2 + 6/5) = (1/2) / (17/10) = 5/17 = 0.29411764705882354
5. Data over noise ratio: p(0) / q(0) = (1/2) / (3/5) = 5/6 = 0.8333333333333334
6. Recovered data probability: 2 * (3/5) * [(5/17) / (1 - 5/17)] = (6/5) * [(5/17) / (12/17)] = (6/5) * (5/12) = 0.5

Experiment

Explore the interactive module to see how the noise distribution affects recovery:

1. Open the Noise-Contrastive Density Ratio module.
2. Set Data distribution to Uneven and Noise distribution to Skewed. Set State probe to 0. Observe the recovered data probability perfectly matches the true probability.
3. Change Noise distribution to Support hole and State probe to 2. Observe that the recovered probability is "Unavailable" because the noise distribution does not cover this state (q(x)=0).

Limitations

This population identity assumes we can find the optimal classifier D(x) exactly. In practice, we estimate D(x) using a neural network and finite data. The recovered density p(x) is only as good as our estimation of D(x). Additionally, NCE requires the noise distribution q(x) to be explicitly known and evaluable, and it must have support everywhere that p(x) does.

References

For the original introduction of Noise-Contrastive Estimation, see: Gutmann, M. and Hyvärinen, A., 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics (pp. 297-304). JMLR Workshop and Conference Proceedings.Link