Recursive Least Squares Forgetting

Update a weighted least-squares solution online while tracking the decay of its prior.

Recursive Least Squares Forgetting

Recursive Least Squares (RLS) algorithms with a forgetting factor allow us to recursively estimate model parameters online while smoothly downweighting older observations. By tracking the decay of its prior, the model continuously adapts to new data, making it particularly suitable for non-stationary environments.

Worked Example

Consider a simple linear model where we iteratively update our parameters θ based on new observations (x_i, y_i). Let the initial parameters be \theta_0 = [0, 0]^T and the initial inverse information matrix P_0 = I / δ (where δ is the initial ridge parameter). Given a forgetting factor λ ∈ (0, 1], at step n, a new observation x_n, y_n arrives. We compute the gain vector K, update the parameter θ, and update the inverse information matrix P as follows:

Compute Gain:K = (P_(n-1) x_n)/(λ + x_n^T P_(n-1) x_n)
Update Parameters:\theta_n = \theta_(n-1) + K(y_n - x_n^T \theta_(n-1))
Update Inverse Information Matrix:P_n = 1/(λ) (P_(n-1) - K x_n^T P_(n-1))

For example, suppose our first data point is x_0 = (1, 0), y_0 = 1 (from Drift data), δ = 1, and λ = 0.9. Initially \theta_0 = [0, 0]^T, P_0 = [[1, 0], [], [0, 1]].
Step 1:
K = ([[1, 0], [], [0, 1]] [[1], [], [0]])/(0.9 + [[1, 0]] [[1, 0], [], [0, 1]] [[1], [], [0]]) = ([[1], [], [0]])/(0.9 + 1) = [[1/1.9], [], [0]] ≈ [[0.5263], [], [0]]
The pre-update residual is 1 - 0 = 1.
\theta_1 = [[0], [], [0]] + [[1/1.9], [], [0]] (1) = [[0.5263], [], [0]]
P_1 = 1/0.9 ( [[1, 0], [], [0, 1]] - [[1/1.9], [], [0]] [[1, 0]] ) = 1/0.9 [[1 - 1/1.9, 0], [], [0, 1]] = [[0.5263, 0], [], [0, 1.1111]]
The decayed prior ridge is 1 × 0.9^1 = 0.9.

Experiment

Interact with the algorithm step-by-step:

Open the Recursive Least Squares Forgetting module.
Choose the Data sequence (e.g., "Drift").
Adjust the Forgetting factor (e.g., 0.9) and Initial ridge.
Change the Steps to observe how the parameters, gains, and the inverse information ellipse update with each new observation. Note how a forgetting factor less than 1 prevents the prior from dominating over time.

Limitations

A pure forgetting factor scales down the entire prior inverse information matrix indiscriminately. If some features are rarely observed, forgetting can cause their corresponding inverse-information eigenvalues to grow unbounded (known as estimator windup). It also assumes a stationary noise variance rather than explicitly modeling parameter drift like a full Kalman filter.

References

The following references provide foundational derivations and analysis for recursive least squares with a forgetting factor.

Recursive Least Squares with Forgetting: Algorithms and Applications.
Provides foundational derivations and analysis of RLS updates and forgetting factors.

Recursive Least Squares Forgetting