Positive Definite Matrices

Positive Definite Matrices and Quadratic Forms

Concept Overview

A symmetric matrix is positive definite if its associated quadratic form always yields a positive value for any non-zero input vector. Geometrically, in two dimensions, the quadratic form of a positive definite matrix creates a paraboloid that opens upwards, with a unique global minimum at the origin. These matrices play a fundamental role in optimization, physics, and machine learning because they guarantee the existence of stable, unique minimums.

Mathematical Definition

An n × n symmetric matrix A is positive definite if for all non-zero vectors x in ℝⁿ:

x^TAx > 0

In 2D, the quadratic form f(x, y) for a symmetric matrix A = [a, b; b, c] is:

f(x, y) = [x, y] [a b] [x] = ax² + 2bxy + cy²

[b c] [y]

Key Concepts

Sylvester's Criterion

A quick way to check if a matrix is positive definite is using Sylvester's Criterion, which states that a symmetric matrix is positive definite if and only if all its leading principal minors are strictly positive. For a 2×2 matrix [a, b; b, c], this means a > 0 and the determinant (ac − b²) > 0.

Eigenvalues

An equivalent definition is that a symmetric matrix is positive definite if and only if all of its eigenvalues are strictly positive (λ_i > 0). If all eigenvalues are positive or zero (λ_i ≥ 0), the matrix is positive semi-definite. If eigenvalues have mixed signs, the matrix is indefinite (creating a saddle point), and if all are negative, it is negative definite.

Quadratic Forms

The expression x^TAx is called a quadratic form. For a 2D matrix, it forms a surface in 3D space. Positive definite matrices create bowl-like shapes (elliptic paraboloids). Indefinite matrices create saddle shapes (hyperbolic paraboloids). The contours (level sets) of a positive definite quadratic form are always ellipses centered at the origin.

Historical Context

The study of quadratic forms and their definiteness dates back to the early 19th century.Carl Friedrich Gauss extensively studied quadratic forms in the context of number theory in his foundational 1801 work Disquisitiones Arithmeticae.

James Joseph Sylvester formalized the conditions for positive definiteness in 1852 with what is now known as Sylvester's Criterion. The concept gained massive practical importance in the 20th century with the rise of mathematical optimization, operations research, and quantum mechanics, where positive definite matrices represent valid metric tensors and observable physical quantities.

Real-world Applications

Machine Learning and Optimization: The Hessian matrix (matrix of second derivatives) of a loss function must be positive definite at a point for that point to be a local minimum.
Statistics: Covariance and correlation matrices must always be positive semi-definite, ensuring that the variance of any linear combination of random variables is non-negative.
Physics and Mechanics: The inertia tensor in rigid body dynamics and the mass matrix in finite element analysis are always positive definite, corresponding to positive kinetic energy.
Control Theory: Lyapunov functions, used to prove the stability of dynamical systems, heavily rely on positive definite matrices (e.g., V(x) = x^TPx).
Computer Graphics: Positive definite matrices define valid metrics for calculating distances and forming ellipses for anisotropic filtering.

Related Concepts

Eigenvalues & Eigenvectors — reveal the definiteness of a matrix
Gradient Descent — optimization relies on traversing a positive definite quadratic bowl
Principal Component Analysis (PCA) — applied to positive semi-definite covariance matrices
Linear Transformations — how symmetric matrices stretch space

Positive Definite Matrices