Bias-Variance Tradeoff

Bias-Variance Tradeoff

Mathematical Definition

Assume we have a true target function y = f(x) + ε, where ε is normally distributed noise with mean 0 and variance σ². We train a model y_hat = f_hat(x) using a training set. The expected squared error of the model at a given point x can be decomposed into three components:

Key Concepts

Bias (Underfitting)

Bias refers to the error introduced by approximating a real-world problem, which may be complex, by a simplified model. High bias models typically make strong assumptions about the data (e.g., assuming a non-linear relationship is strictly linear). High bias leads to underfitting, where the model fails to capture the relevant relations between features and target outputs.

Variance (Overfitting)

Variance is the amount that the model's predictions change if we train it on a different dataset. A high variance model pays too much attention to the training data, capturing the random noise as if it were a true pattern. This leads to overfitting, resulting in low training error but high test error. In our interactive visualization, high polynomial degrees exhibit high variance, with the fitted curves changing drastically for different training sets.

Irreducible Error

Irreducible error (σ²) is the noise inherent in the problem itself. This error cannot be reduced by choosing a better model or algorithm. It arises from unknown variables that influence the target but are not captured by the features, or fundamental randomness in the process being modeled.

The Tradeoff

As you increase model complexity (e.g., adding more parameters or increasing polynomial degree), bias decreases because the model can fit the true underlying function better. However, variance increases because the model becomes more sensitive to the specific noise in the training set. Decreasing complexity reduces variance but increases bias. Finding the optimal model requires balancing these two competing forces to minimize total expected error.

Historical Context

The bias-variance tradeoff decomposition was formalized in the context of neural networks by Geman, Bienenstock, and Doursat in their seminal 1992 paper "Neural Networks and the Bias/Variance Dilemma". While the concepts of overfitting and underfitting had long been known in statistics, their formulation provided a rigorous mathematical framework for understanding why more complex neural networks do not always yield better generalization.

Recently, the classic "U-shaped" bias-variance curve has been challenged by the phenomenon of "double descent" in deep learning, where extremely over-parameterized models (like modern Large Language Models) seem to defy the tradeoff, achieving lower test error even as variance theoretically should increase. This remains an active area of research in statistical learning theory.

Real-world Applications

• Model Selection: Choosing the right degree for polynomial regression or the appropriate maximum depth for a Decision Tree.
• Regularization: Techniques like L1 (Lasso) and L2 (Ridge) regression deliberately introduce a small amount of bias to significantly reduce variance.
• Ensemble Methods: Random Forests reduce variance by averaging the predictions of many high-variance, low-bias decision trees. Boosting algorithms like Gradient Boosting reduce bias by sequentially combining high-bias, low-variance weak learners.
• Hyperparameter Tuning: Cross-validation is used empirically to find the hyperparameter values that hit the optimal tradeoff point.

Related Concepts

• Cross-Validation — empirical technique to estimate generalization error and find the bias-variance sweet spot.
• Linear Regression — a naturally high-bias, low-variance model.
• Decision Tree — a non-parametric model prone to low bias but very high variance without pruning.