Rayleigh Quotient

Mathematical Definition and Critical Points

For a real symmetric n × n matrix A and a non-zero vector x ∈ ℝⁿ, the Rayleigh quotient R(A, x) is defined as:

R(A, x) = (xᵀ A x) / (xᵀ x)

Because R(A, cx) = R(A, x) for any non-zero scalar c, the Rayleigh quotient is scale-invariant and can be studied exclusively on the unit sphere ‖x‖ = 1, where it simplifies to the quadratic form R(A, x) = xᵀ A x.

Taking the gradient with respect to x yields:

∇ R(A, x) = 2 · (A x − R(A, x) x) / (xᵀ x)

Setting ∇ R(A, x) = 0 reveals that the stationary points of the Rayleigh quotient are precisely the eigenvectors of A, and the corresponding stationary values are the eigenvalues: A x = λ x implies R(A, x) = λ.

Worked example: 2 × 2 symmetric matrix and Courant-Fischer bounds

Consider the symmetric matrix A = [[3, 1], [1, 3]] and parameterize trial vectors on the unit circle by angle θ ∈ [0, 2π):

x(θ) = [cos θ, sin θ]ᵀ

1. Compute the matrix trace and determinant: tr(A) = 3 + 3 = 6, and det(A) = 3 × 3 − 1 × 1 = 8.
2. Solve the characteristic equation: λ² − tr(A)λ + det(A) = λ² − 6λ + 8 = (λ − 4)(λ − 2) = 0. The eigenvalues are λ₁ = 4.0 and λ₂ = 2.0.
3. Determine the corresponding normalized eigenvectors: For λ₁ = 4: (A − 4I)v = 0 gives −v₁ + v₂ = 0, so v₁ = [1, 1]ᵀ / √2, corresponding to θ = 45° (or 225°). For λ₂ = 2: (A − 2I)v = 0 gives v₁ + v₂ = 0, so v₂ = [−1, 1]ᵀ / √2, corresponding to θ = 135° (or 315°).
4. Expand the Rayleigh quotient as a function of angle θ:
   R(A, x(θ)) = [cos θ, sin θ] [[3, 1], [1, 3]] [cos θ, sin θ]ᵀ
   = 3 cos² θ + 2 sin θ cos θ + 3 sin² θ = 3 + sin(2θ).
5. Analyze the extreme values: Since −1 ≤ sin(2θ) ≤ 1, the maximum occurs when sin(2θ) = 1 (2θ = 90°, so θ = 45°), yielding R(A, x) = 3 + 1 = 4.0 = λ_max. The minimum occurs when sin(2θ) = −1 (2θ = 270°, so θ = 135°), yielding R(A, x) = 3 − 1 = 2.0 = λ_min.
6. Verify Courant-Fischer Min-Max Theorem: For every unit trial direction θ, the Rayleigh quotient satisfies 2.0 ≤ R(A, x(θ)) ≤ 4.0.

Notice that at θ = 0° (x = [1, 0]ᵀ), R(A, x) evaluates to exactly 3.00. Applying one step of Rayleigh quotient iteration (RQI) by solving (A − 3I)y = x yields y = [1, 1]ᵀ, which normalizes directly to the true dominant eigenvector [1/√2, 1/√2]ᵀ, demonstrating the cubic local convergence characteristic of RQI.

Interactive experiment: exploring stationary points and bounds

Open the Rayleigh Quotient module. The module provides sliders for Matrix Entry a11, Matrix Entry a22, Off-diagonal Entry a12/a21, Vector Angle (theta), and Iteration Steps (k).

1. Set Matrix Entry a11 to 3, Matrix Entry a22 to 3, and Off-diagonal Entry a12/a21 to 1. Confirm that the calculated eigenvalue bounds display λ_max = 4.00 and λ_min = 2.00.
2. Adjust Vector Angle (theta) to 45°. Observe that the Rayleigh quotient readout reaches exactly 4.00, and the status indicator reports "Stationary (Eigenvector)". Notice that the mapped vector Ax points in the exact same direction as x with magnitude scaling 4.0.
3. Rotate Vector Angle (theta) to 135°. Observe that the Rayleigh quotient drops to 2.00, confirming the lower Courant-Fischer bound. Ax is once again collinear with x, scaled by λ = 2.0.
4. Set Vector Angle (theta) to 0° (where R(A, x) = 3.00). Now increase Iteration Steps (k) from 0 to 1. Observe that the RQI vector jumps directly to 45° with quotient 4.00, demonstrating rapid local convergence.
5. Test the boundary zero matrix by setting a11 = 0, a22 = 0, and a12/a21 = 0. Verify that for any angle (e.g. θ = 60°), R(A, x) evaluates cleanly to 0.00 without numerical overflow or divide-by-zero errors.

Mathematical limitations and numerical considerations

The classical Courant-Fischer Min-Max theorem strictly requires that matrix A be Hermitian (or real symmetric), ensuring real eigenvalues and an orthonormal basis of eigenvectors. For non-symmetric matrices, eigenvalues may be complex, eigenvectors may not be orthogonal, and the Rayleigh quotient xᴴ A x / xᴴ x is not generally real-valued.

Additionally, while Rayleigh quotient iteration exhibits cubic convergence asymptotically near simple eigenvalues, if the initial vector x is chosen exactly orthogonal to an eigenspace or near an unstable saddle point, numerical roundoff or ill-conditioned shifted matrix solves (A − σI) can induce momentary stagnation before roundoff breaks the symmetry.

Primary sources and educational references

Strang, G. (2016). Introduction to Linear Algebra (5th ed.). Wellesley-Cambridge Press covers symmetric positive definite matrices, Rayleigh quotients, and the Min-Max theorem in Chapter 6.

Golub, G. H., & Van Loan, C. F. (2013). Matrix Computations (4th ed.). Johns Hopkins University Press details Rayleigh quotient iteration, invariant subspaces, and cubic convergence proofs in Chapter 8.

Horn, R. A., & Johnson, C. R. (2012). Matrix Analysis (2nd ed.). Cambridge University Press provides the rigorous variational formulation of Courant-Fischer characterizations and Rayleigh-Ritz methods in Chapter 4.