Gauss–Newton Residual Curvature

Separate positive Gauss–Newton curvature from the residual term in the true Hessian.

Gauss–Newton Residual Curvature

In nonlinear least-squares optimization, the objective function is the sum of squared residuals, f(x) = 0.5 * ||r(x)||^2. The true Hessian matrix of this objective contains two terms: the Gauss–Newton matrix G = J^T J, which is always positive semi-definite and only depends on the first derivatives of the residuals, and a second term that involves the second derivatives of the residuals weighted by the residuals themselves. Separate positive Gauss–Newton curvature from the residual term in the true Hessian.

Worked Example: Curvature at (0, 0)

Consider the residual model r1 = x² + y - 1 and r2 = x + y² - 1. Let's calculate the Gauss-Newton and true Hessian matrices at the point (0, 0) with zero damping.

1. Point x = 0, Point y = 0
2. Calculate residuals:
   r1 = 0² + 0 - 1 = -1
   r2 = 0 + 0² - 1 = -1
3. Calculate Jacobian J = [[2x, 1], [1, 2y]]:
   J = [[0, 1], [1, 0]]
4. Calculate Gauss-Newton matrix G = J^T J:
   G = [[0, 1], [1, 0]] * [[0, 1], [1, 0]] = [[1, 0], [0, 1]]
   The minimum eigenvalue of G is 1.0.
5. Calculate true Hessian H = G + diag(2*r1, 2*r2):
   H = [[1, 0], [0, 1]] + [[-2, 0], [0, -2]] = [[-1, 0], [0, -1]]
   The minimum eigenvalue of H is -1.0.
6. Calculate Gradient = J^T r:
   Grad = [[0, 1], [1, 0]] * [-1, -1]^T = [-1, -1]^T
7. Proposed GN step = -G^-1 Grad:
   Step = -[[1, 0], [0, 1]] * [-1, -1]^T = [1, 1]
8. Proposed Newton step = -H^-1 Grad:
   Step = -[[-1, 0], [0, -1]] * [-1, -1]^T = [-1, -1]

Experiment: Exploring Curvature Differences

We can observe how the Gauss-Newton approximation differs from the true Hessian using the interactive module.

Open the Gauss–Newton Residual Curvature module.
Set Point x to 0, Point y to 0, and Damping to 0.
Observe that the GN minimum eigenvalue is 1.0, while the True Hessian minimum eigenvalue is -1.0. The Gauss-Newton matrix is positive definite, but the true Hessian is negative definite, indicating a local maximum despite a zero gradient.
Set Point x to 0.5, Point y to 0.5, and Damping to 0. The Gauss-Newton matrix becomes singular (minimum eigenvalue 0.0), making the GN step x and GN step y Unavailable.
Set Damping to 1 to add regularization. Now, both the damped Gauss-Newton matrix and the damped true Hessian become invertible, and the steps can be computed.

Limitations

This model uses a specific, simple residual function r1 = x² + y - 1, r2 = x + y² - 1 to illustrate the difference between the Gauss-Newton and true Hessian matrices. In practice, full Newton or Gauss-Newton steps are just proposals for step directions; they do not guarantee descent or convergence without line search or trust region methods. Additionally, singular matrices lead to unavailable step coordinates since arbitrary pseudo-inverses are not used here.

References

For further reading and mathematical foundations, please refer to the following primary resource which explains the Gauss-Newton approximation, the role of residuals in the true Hessian, and the implications for convergence in nonlinear least squares problems.

Cornell CS4220: Nonlinear Least Squares and Gauss-Newton

Gauss–Newton Residual Curvature