Locally Weighted Linear Regression
Move a local weighted fit along a curved dataset and inspect the bandwidth tradeoff.
By Riano · Published · Updated · Report a correction
Locally Weighted Linear Regression
Locally Weighted Linear Regression (LOESS or LOWESS) is a non-parametric regression method that combines multiple regression models in a k-nearest-neighbor-based meta-model. Instead of fitting a single global line to the entire dataset, it computes a separate linear fit for every target query point using only the points closest to it, weighted by distance.
Worked Example
Consider fitting a local line for a query point at x = 0 with bandwidth h = 1 using a dataset of fixed points [-2, -1, 0, 1, 2] corresponding to y = x2.
The Gaussian weights are given by w = exp(-(x - query)2 / (2h2)). For the points, we get weights approximately:
x = -2:w = exp(-(-2)2 / 2) ≈ 0.135x = -1:w = exp(-(-1)2 / 2) ≈ 0.607x = 0:w = exp(-(0)2 / 2) = 1.000x = 1:w = exp(-(1)2 / 2) ≈ 0.607x = 2:w = exp(-(2)2 / 2) ≈ 0.135
Sum of weights = 0.135 + 0.607 + 1.000 + 0.607 + 0.135 ≈ 2.484.
The local weighted normal equations for a + b*(x-q) involvesum(w*(x-q)), sum(w*(x-q)2), sum(w*y), sum(w*y*(x-q)). By solving these explicit 2x2 weighted normal equations:
a = 0.9243121604036103 b = 0.0
We get a ≈ 0.924 as our local prediction.
Experiment
To understand the effect of bandwidth on local predictions in Locally Weighted Linear Regression:
- Keep Query x at exactly
0. - Set Bandwidth to exactly
0.5. Note the small Effective sample size and a prediction close to 0 (the value of the origin). - Change Bandwidth to exactly
1. Notice how points further away gain more weight and drag the prediction upward. - Change Bandwidth to exactly
2. The prediction rises further as the distant high-y points heavily influence the local fit.
Limitations
Locally weighted linear regression requires keeping the entire dataset in memory to make predictions, meaning prediction time is proportional to dataset size rather than a fixed model size. It is also sensitive to the choice of bandwidth; too large a bandwidth makes the fit too rigid and global, while too small a bandwidth can cause overfitting or lead to singular normal equations if few points carry non-zero weight.
References
The handbook explains LOESS/LOWESS locally weighted scatterplot smoothing, describing the process of fitting local low-degree polynomials with a weighting function that decays as distance increases, and addressing bandwidth selection tradeoffs.
Experience it interactively
Adjust parameters, observe in real time, and build deep intuition with Riano’s interactive Locally Weighted Linear Regression module.
Try Locally Weighted Linear Regression on Riano →