Tweedie Gaussian Denoising

Worked Example

Let the latent variable X₀ be drawn from a Gaussian prior with mean μ = 0 and variance v = 1. We observe Y = X₀ + ε, where the noise ε is independent and drawn from a Gaussian with zero mean and variance n = 1. Suppose we observe y = 2.

The marginal distribution of Y is the convolution of the prior and the noise distribution. Since both are Gaussian, Y is distributed as Normal(μ, v + n). Here, Y ~ Normal(0, 2).

The marginal score is defined as the derivative of the log-marginal density of Y with respect to y. For a Gaussian, the score is simply -(y - μ) / (v + n). Thus:

Score = -(2 - 0) / (1 + 1) = -2 / 2 = -1.0

Tweedie's formula gives the posterior expectation E[X₀ | Y = y] as y + n * Score. We can calculate this directly:

Posterior Mean = 2 + 1 * (-1.0) = 2 - 1.0 = 1.0

The Tweedie correction term is exactly n * Score = -1.0. This pulls the observed value y = 2 toward the prior mean μ = 0 by exactly 1.0 unit.

The posterior variance for conjugate Gaussians is independent of the observation y and is given by (v * n) / (v + n):

Posterior Variance = (1 * 1) / (1 + 1) = 0.5

Experiment

You can interact with these variables directly in the module: Tweedie Gaussian Denoising. The visualization shows how the densities change and visualizes the Tweedie correction as a yellow arrow.

1. Set the Prior mean to 0, Prior variance to 1, Noise variance to 1, and Observed value to 2. Observe the Marginal score of -1.0 and Posterior mean of 1.0, matching our worked example.
2. Change the Prior variance to 0.25 and the Noise variance to 4. Notice the marginal distribution becomes much wider, and the posterior mean is strongly pulled towards the prior, yielding approximately 1.058824.
3. Set the Noise variance to 0. The marginal score becomes -2.0, but because the noise is zero, the correction is 0.0. The posterior perfectly recovers the observation y = 2, and the posterior variance is 0.
4. Set both Prior variance and Noise variance to 0. Set Observed value to 0. The marginal distribution is a consistent point mass, and the log-density score is unavailable.

Limitations

While Tweedie's formula is exact for exponential families, there are important practical and mathematical limitations when applied.

• Degenerate Support: When both prior and noise variances are zero, the marginal distribution collapses to a Dirac delta point mass. In this setting, an ordinary log-density score does not exist. If the observation differs from the prior mean, the event has zero probability and the posterior is fundamentally undefined.
• Empirical Score Estimation: In modern diffusion models and score matching, the true marginal score is unknown and must be estimated via a neural network (e.g., a UNet). The quality of the Tweedie denoising step is therefore entirely dependent on the accuracy of this score approximation.
• Model Misspecification: If the true latent prior or noise distribution deviates significantly from Gaussian (or the chosen exponential family), the linear score form -(y - μ) / (v + n) no longer holds, and more complex score functions are required.

References

The original formulation and applications of Tweedie's formula provide the foundation for modern empirical Bayes and score-based generative modeling.

• Efron, B. (2011). Tweedie's Formula and Selection Bias. Journal of the American Statistical Association, 106(496), 1602-1614.