Oja Hebbian Principal Direction Flow

The Oja learning rule modifies the classical Hebbian rule by adding a weight decay term proportional to the square of the output, preventing weights from growing indefinitely and stabilizing them near a principal direction of the input data covariance matrix.

Worked Example

Let the initial weight vector be w = (0.866025, 0.5) (angle 30°, radius 1). The covariance matrix C has eigenvalues 3 and 1 with rotation 0, so C = diag(3, 1). We apply a single Euler step with rate η = 0.1.

First compute Cw = (3 * 0.866025, 1 * 0.5) = (2.598076, 0.5).
Next, find the output squared: (w^TCw) = 0.866025 * 2.598076 + 0.5 * 0.5 = 2.25 + 0.25 = 2.5.
The standard Hebbian update is w_new = w + ηCw = (0.866025 + 0.2598, 0.5 + 0.05) = (1.1258, 0.55). Its squared norm is ~1.57.
The Oja update is w_new = w + η(Cw - (w^TCw)w) = w + η( (2.598076, 0.5) - 2.5 * (0.866025, 0.5) ) = w + η(2.598076 - 2.16506, 0.5 - 1.25) = w + η(0.43301, -0.75) = (0.866025 + 0.0433, 0.5 - 0.075) = (0.90933, 0.425).
The squared norm of the Oja step is 0.90933² + 0.425² = 0.82688 + 0.180625 = 1.0075.

Experiment

Explore how the single-step update direction depends on the initial state and the covariance geometry by using the Oja Hebbian Principal Direction Flow module.

Change the Weight angle to observe how the update vector (the Oja step) pulls the weight vector towards the dominant principal axis (the major eigenvector).
Set the Covariance rotation to 45 to see how the axes of attraction change correspondingly.
Set the Euler step to 0.25 to see how larger learning rates increase the numerical divergence from the unit circle, even though the continuous ODE preserves it.
Adjust the Weight radius to 0.5 or 2 to see the effect on the continuous norm derivative and how the rule either grows or shrinks the vector back toward norm 1.

Limitations

This module models a simplified mean-field one-neuron scenario representing the averaged learning dynamics over many data samples, rather than stochastic updates on individual points. While the continuous Oja ODE exactly preserves the unit norm if started on it, discrete Euler integration steps naturally accumulate finite second-order errors, causing the vector to slightly drift off the unit sphere. The visualization here shows a 2D projection; higher-dimensional generalization requires multi-neuron rules (like Sanger's rule) to extract multiple orthogonal components.

References

The original formulation of this single-neuron stabilizing learning rule for principal component analysis.

Oja, E. (1982). A simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15(3), 267-273. https://doi.org/10.1007/BF00275687

Oja Hebbian Principal Direction Flow