Orthogonal Projections

Orthogonal Projections

Mathematical Definition

Projection onto a Line (1D Subspace)

Let W be a 1-dimensional subspace spanned by a non-zero vector v. The orthogonal projection of a vector x onto W is given by:

This is algebraically equivalent to using the dot product formula: [(x · v) / (v · v)] v.

Projection Matrix (1D Subspace)

We can express this operation as a linear transformation P (the projection matrix) applied to x. By rearranging the formula, we factor out x:

Projection onto General Subspaces

If W is an m-dimensional subspace (where m > 1) spanned by the linearly independent columns of a matrix A, the projection matrix P becomes more generalized:

Notice that when A is just a single column vector v, A^TA is a scalar (the dot product v^Tv), and its inverse is just division by that scalar, reducing perfectly to the 1D formula.

Key Concepts

Idempotence

A defining property of any projection matrix is that projecting twice is the same as projecting once: P² = P. Once a vector is projected onto a subspace, it is already entirely within that subspace. Projecting it again changes nothing.

Symmetry

For an orthogonal projection, the projection matrix is symmetric: P = P^T. (Note that non-orthogonal, or "oblique", projections exist where P is not symmetric, but orthogonal projections are strictly symmetric).

The Error Vector (Orthogonality Principle)

The difference between the original vector x and its projection Px is called the error vector e = x - Px. The defining geometric feature of an orthogonal projection is that e is perfectly orthogonal to the subspace W. This means the dot product of e with any vector in W is zero.

Historical Context

The formalized algebraic method of orthogonal projections was largely developed in the 19th century alongside the maturation of linear algebra. Carl Friedrich Gauss and Adrien-Marie Legendre famously used these concepts to formulate the method of Least Squares around the turn of the 19th century to predict planetary and cometary orbits by projecting noisy astronomical observation vectors onto the subspace defined by theoretical models.

Later, the geometric interpretation of matrices as linear transformations, pioneered by Arthur Cayley and James Joseph Sylvester, unified the algebraic formulas of projection with their geometric shadows.

Real-world Applications

• Least Squares Regression: In statistics and machine learning, when finding the line of best fit for data that cannot be solved perfectly (an overdetermined system Ax = b), we project the data vector b onto the column space of A to find the closest possible solution.
• Computer Graphics: Rendering 3D scenes onto a 2D monitor inherently requires projecting 3D vectors onto a 2D plane (though often perspective projections are used rather than purely orthogonal ones).
• Data Compression (PCA): Principal Component Analysis projects high-dimensional data onto a lower-dimensional subspace (the principal components) while minimizing the orthogonal error (information loss).
• Signal Processing: Signals can be projected onto a basis of orthogonal functions (like sine and cosine waves in the Fourier Transform) to filter noise or compress audio.

Related Concepts

• Dot Product: The fundamental algebraic building block used to calculate projections.
• Least Squares Approximation: The direct application of projecting a vector onto the column space of a matrix to solve overdetermined systems.
• Gram-Schmidt Process: An algorithm that uses orthogonal projections sequentially to turn any basis of a subspace into an orthogonal basis.