Hessian–Vector Directional Propagation

Propagate a direction through a gradient graph to reveal curvature without a dense Hessian.

Hessian–Vector Directional Propagation

In many machine learning applications, computing the full Hessian matrix (second derivatives) is too expensive. The Pearlmutter forward-over-reverse method allows propagating a specific direction through a gradient computation graph to reveal local curvature without forming a dense Hessian.

Worked Example

Let's calculate the propagation of the X direction vector e1=(1,0) at the point (0,1) for the Rosenbrock function f(x,y) = (1-x)² + 100(y-x²)².

Given f(x,y) = (1-x)² + 100(y-x²)², at x=0, y=1: Objective: f(0,1) = (1-0)² + 100(1-0²)² = 1 + 100 = 101 Gradient: ∇f(x,y) = [ -2(1-x) - 400x(y-x²), 200(y-x²) ] ∇f(0,1) = [ -2(1) - 400(0)(1), 200(1) ] = [ -2, 200 ] Hessian: H_11 = 2 - 400(y-x²) + 800x² = 2 - 400(1) + 0 = -398 H_12 = -400x = 0 H_21 = -400x = 0 H_22 = 200 H(0,1) = [ -398,   0 ] [    0, 200 ] Propagation of direction v = (1,0): H * v = [ -398 * 1 + 0 * 0, 0 * 1 + 200 * 0 ] = [ -398, 0 ] Directional curvature: v^T H v = 1 * (-398) + 0 * 0 = -398 Directional first derivative: ∇f(0,1) · v = -2 * 1 + 200 * 0 = -2

Experiment

You can interact with the directional propagation in the interactive module.

Set Point x to 1.
Set Point y to 1.
Set Direction to Diagonal.

Observe that at (1,1), the gradient is [0, 0] since it is the global minimum. The directional curvature is 202, showing positive curvature in this diagonal direction.

Limitations

This method assumes that the gradient graph is differentiable and that the direction vector is properly scaled. It evaluates curvature along a single, fixed direction rather than analyzing the entire eigenspectrum of the Hessian, which means saddle points might not be fully identified if the wrong direction is chosen. At (0,1), the X direction gives a curvature of -398 despite the objective function being non-negative everywhere, demonstrating that we cannot assume the Hessian is positive semi-definite (PSD) far from the minimum.

References

The original Pearlmutter method is detailed in the following reference.

Fast Exact Multiplication by the Hessian - B. A. Pearlmutter (1994), Neural Computation.

Hessian–Vector Directional Propagation