Multivariate Normal Distribution

Visualize 2D normal distributions with varying correlation and standard deviations.

Multivariate Normal Distribution

Concept Overview

The multivariate normal distribution is a multidimensional generalization of the one-dimensional (univariate) normal distribution. While a univariate normal distribution describes the probability distribution of a single random variable, the multivariate normal distribution describes the joint probability distribution of two or more random variables. It is fundamentally defined by its mean vector and its covariance matrix, capturing not only the variance of each individual variable but also how the variables covary with one another.

Mathematical Definition

A random vector X = (X₁, ..., X_k)^T is said to have a k-variate normal distribution if its probability density function is given by:

f(x) = (2π)^-k/2 |Σ|^-1/2 exp(-1/2 (x - μ)^T Σ^-1 (x - μ))

Where:

x is a real k-dimensional column vector.
μ is the k-dimensional mean vector, E[X].
Σ is the k × k covariance matrix, E[(X - μ)(X - μ)^T].
|Σ| is the determinant of Σ.

For the bivariate case (k=2) shown in the visualization, the covariance matrix Σ is defined by the standard deviations σ_X, σ_Y, and the correlation coefficient ρ.

Key Concepts

Covariance Matrix (Σ): This matrix dictates the shape and orientation of the distribution. Diagonal elements represent individual variances (σ²), while off-diagonal elements represent covariances. It must be symmetric and positive-semi-definite.
Correlation (ρ): A normalized measure of covariance, ranging from -1 to 1. When ρ = 0, the variables are uncorrelated, and the contours of the distribution align with the coordinate axes. As ρ approaches 1 or -1, the distribution stretches into an elliptical shape diagonally.
Confidence Ellipses: The level sets (contours of equal probability density) of a multivariate normal distribution form ellipsoids. In 2D, these are ellipses where points within a certain boundary lie within a specific Mahalanobis distance from the mean, representing regions of given probability.
Marginal Distributions: An important property is that any subset of variables from a multivariate normal distribution also follows a normal distribution (the marginal distribution).

Historical Context

The origins of the normal distribution trace back to Abraham de Moivre in the 18th century, and later to Carl Friedrich Gauss and Pierre-Simon Laplace. The extension to multiple dimensions developed significantly in the late 19th and early 20th centuries. Francis Galton's work on correlation and regression in the 1880s provided a conceptual foundation for bivariate relationships. Karl Pearson mathematically formalized these concepts, leading to the multivariate normal distribution as a central tool in statistical analysis, particularly for the development of principal component analysis (PCA) and multivariate analysis of variance (MANOVA).

Real-world Applications

Machine Learning: Fundamental to algorithms like Gaussian Mixture Models (GMMs) for clustering and Gaussian Naive Bayes for classification.
Finance: Used in modern portfolio theory to model the joint returns of multiple assets, where covariance captures how asset prices move together.
Robotics and Tracking: Kalman filters rely heavily on multivariate normal distributions to track the state of moving objects by combining noisy sensor measurements.
Quality Control: Monitoring multiple correlated manufacturing metrics simultaneously to detect anomalies that might not be obvious when looking at each metric individually.

Related Concepts

Probability Distributions — The foundation for both univariate and multivariate forms.
Correlation & Covariance — The building blocks of the covariance matrix.
Copulas & Joint Distributions — Advanced methods for modeling joint distributions beyond the strict assumptions of multivariate normality.

Multivariate Normal Distribution