Quadratic Discriminant Gaussian Boundary

Quadratic Discriminant Analysis (QDA) is a generative model used for classification. Unlike Linear Discriminant Analysis (LDA), which assumes all classes share the same covariance matrix, QDA estimates a separate covariance matrix for each class.

Worked Example

Let's calculate the log-odds for a specific point where the classes have different covariance matrices. Assume Class A has mean \mu_A = [-1, 0] and Class B has mean \mu_B = [1, 0]. Their covariance matrices are \Sigma_A = diag(0.25, 1) and \Sigma_B = diag(1, 0.25). Let the priors be equal, so P(A) = P(B) = 0.5, and we are evaluating at point x = [0, 0].

The difference vectors are x - \mu_A = [1, 0] and x - \mu_B = [-1, 0].
The inverse covariance matrices are \Sigma_A⁻¹ = diag(4, 1) and \Sigma_B⁻¹ = diag(1, 4).

The Mahalanobis distances are:

Mahalanobis A = [1, 0] * diag(4, 1) * [1, 0]^T = 4 Mahalanobis B = [-1, 0] * diag(1, 4) * [-1, 0]^T = 1

The determinants are |\Sigma_A| = 0.25 and |\Sigma_B| = 0.25.
We evaluate the log-odds:
```
Log Odds(A/B) = -0.5 * (4 - 1) = -1.5
```

The negative log odds (-1.5) indicate that Class B is more probable at the origin, with a posterior of about 81.8%.

Experiment

You can test the decision boundary curvature and point evaluations in the interactive module:Quadratic Discriminant Gaussian Boundary.

Set the Covariances control to "Equal". Observe that the boundary becomes a straight line, as the quadratic terms cancel out (coefficients become 0).
Change the Covariances to "Different". The boundary is now curved (a hyperbola or ellipse) due to the distinct class variances.
Set the Prior A to "0.75" and observe how the decision boundary shifts to favor Class A.
Adjust Point x and Point y to explore different regions of the feature space and see how the posterior probabilities update.

Limitations

QDA requires fitting a full covariance matrix for each class, which means it needs more parameters than LDA. In high-dimensional spaces with few training samples, estimating these matrices can lead to overfitting or singular matrices. Additionally, QDA strictly assumes that the data in each class is generated by a multivariate Gaussian distribution; it may perform poorly if the actual data distribution is highly skewed or multi-modal. When a prior is exactly 0 or 1, the log-odds become infinite, and the posterior is completely deterministic.

References

For more details on Quadratic Discriminant Analysis and its comparison with LDA, refer to thescikit-learn documentation on LDA and QDA, which covers the mathematical formulation and assumptions.

Quadratic Discriminant Gaussian Boundary