Gaussian Process Regression

Worked Example

Consider fitting a Gaussian Process to fixed training pairs (0, 0) and (1, 1). We use a zero prior mean and a unit-amplitude Radial Basis Function (RBF) kernel defined as k(x,z) = exp(-(x-z)^2 / (2 * ell^2)), where ell is the length scale.

Let ell = 1 and the observation noise variance be 0. The covariance matrix K of our training inputs is a 2x2 matrix with 1s on the diagonal and off-diagonal entries r = exp(-1/2) approx 0.6065. Thus, the system matrix A = K + noiseVariance * I is precisely equal to K.

The inverse of a 2x2 matrix A is given by A^-1 = 1/(ad-bc) * [d, -b; -c, a]. Applying this, A^-1 = 1/(1 - r^2) * [1, -r; -r, 1].

To compute the posterior mean and latent variance at a query point x_* = 0.5, we first construct the cross-covariance vector k_* = [k(x_*, x_1), k(x_*, x_2)]^T = [exp(-0.5^2/2), exp(-0.5^2/2)]^T approx [0.8825, 0.8825]^T.

The posterior mean is given by mu_* = k_*^T * A^-1 * y = k_*^T * A^-1 * [0, 1]^T = 1/(1 - r^2) * [0.8825, 0.8825] * [-r, 1]^T approx 0.549318. The displayed value rounds to six decimal places; the inverse retains unrounded values.

The latent variance (i.e., uncertainty of the underlying noise-free function) is computed as sigma_*^2 = k(x_*, x_*) - k_*^T * A^-1 * k_* = 1 - k_*^T * A^-1 * k_* approx 0.030456. This small but nonzero variance follows from the chosen kernel and two noise-free observations; it is not a general guarantee between observed points.

1. Calculate K covariance matrix elements: K(0,1) = 0.6065
2. Invert A = K + noise: A^-1 = [1/(1-0.6065^2)] * [1, -0.6065; -0.6065, 1]
3. Calculate posterior mean at x=0.5: mu = 0.549318

Experiment

This visualization allows you to explore the effect of GP hyperparameters on the predictive distribution. View the module here: Gaussian Process Regression.

1. Set Length scale to 1, Noise variance to 0 and Query x to 0.5. Read mean 0.549318, latent variance 0.030456 and standard deviation 0.174518.
2. Set Noise variance to 1. The matrix diagonal becomes 2, mean becomes 0.338571 and latent variance becomes 0.402423. The variance control is added once, not squared.
3. Restore noise to 0 and set Length scale to 0.5. At the midpoint, read mean 0.534230 and latent variance 0.351946.
4. Set Query x to 0: the noise-free training point has mean and variance zero. Set length scale to 2 and query to 1: read mean one and variance zero.
5. Drag the plot or use arrow keys, then Home or Reset view. View changes preserve the matrix and posterior readouts.

Limitations

While exact function-space Bayesian inference is extremely powerful, it is computationally expensive.

• Fixed Data and Scale: This visualization uses a fixed training set and a basic 2x2 GP inverse. In practice, calculating the exact posterior for n data points requires O(n^3) time for matrix inversion. For large datasets, exact GPs are intractable, necessitating approximations like sparse GPs (e.g. using inducing points).
• Deterministic Band: The plotted uncertainty band represents the theoretical variance of the latent function conditional strictly on the provided hyperparameter values. It is not an optimized maximum marginal likelihood learned model, nor is it a frequentist confidence interval derived through random sampling.
• Roundoff Clamping: Because computers use finite floating-point precision, calculating 1 - k_*^T * A^-1 * k_* very near training points under low noise conditions can result in microscopically negative values. Only negative roundoff no smaller than -1e-12 is clamped to zero. A more negative or nonfinite result is reported as a numerical error rather than a fabricated zero variance.

References

Review these resources to understand exactly how Gaussian Processes are formulated and scaled.

• Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian Processes for Machine Learning (Chapter 2).
• scikit-learn Developers. Gaussian Processes.