Linear Discriminant Analysis

Find a class-aware projection by balancing mean separation against within-class covariance.

Linear Discriminant Analysis

Linear Discriminant Analysis (LDA) is a method to find a linear combination of features that characterizes or separates two or more classes of objects or events. The resulting combination may be used as a linear classifier or for dimensionality reduction.

Worked Example

Consider two Gaussian classes with equal prior probabilities:

Class 0: mean μ₀ = (-1, 0)
Class 1: mean μ₁ = (1, 0)
Common covariance matrix: Σ =
[1, 0.5]
[0.5, 1]

This corresponds to a correlation of 0.5 with no shrinkage. To find the decision boundary, we first determine the Fisher direction vector w, which maximizes the separation between the projected means while minimizing the within-class variance.

w is proportional to Σ^-1(μ₁ - μ₀).

Calculate det(Σ) = 1 - 0.5² = 0.75.
Calculate Σ^-1 = (1/0.75) * [[1, -0.5], [-0.5, 1]].
The difference in means is μ₁ - μ₀ = (2, 0).
Multiplying gives Σ^-1 * (2, 0) = (1/0.75) * (2, -1).
Normalizing this vector (with a positive x sign) gives w ≈ (0.894, -0.447).
Fisher separation J = 4 / (1 - 0.5²) = 5.333333333333333.
For a probe point x = 1, y = 0, the log likelihood ratio is 2*(1 - 0.5*0) / (1 - 0.5²) = 2.6666666666666665. The threshold is 0, so the expected result is predicted class 1.

Experiment

Try the Linear Discriminant Analysis module.

Set Correlation to 0.5.
Set Shrinkage to 0.
Set Probe x to 1 and Probe y to 0.
Observe the Fisher separation (5.333) and log likelihood ratio (2.667). The direction vector is normalized. The predicted class is 1.
Change Probe x to 0 and Probe y to 1. The log likelihood ratio becomes -1.333, and the predicted class changes to 0.
Change Correlation to -0.5. The direction vector's y component flips sign, and the covariance ellipses tilt in the opposite direction.
Set Shrinkage to 1. The covariance is shrunk to the identity matrix, turning the ellipses into circles and decoupling the dimensions.

Limitations

LDA assumes that the classes are normally distributed and share the same covariance matrix. When these assumptions are violated (e.g., heavily skewed data, or different covariances requiring Quadratic Discriminant Analysis), LDA may perform poorly. In this specific module, the threshold is fixed at 0 for equal priors.

References

scikit-learn Documentation: Linear and Quadratic Discriminant Analysis. This provides a theoretical and practical overview of LDA, including its derivation and use as a classifier and dimensionality reduction technique.

These references provide the theoretical foundations for the assumptions and formulations used in LDA.

Linear Discriminant Analysis