Dual Number Forward Differentiation

Worked example

Consider the function f(x) = x³ - 2x + 1. We want to evaluate the function and its derivative at x = 0.5. Instead of relying on symbolic differentiation or finite differences, we use dual numbers.

A dual number is represented as a pair (v, d), where v is the real value and d is the dual part. The fundamental property of the dual unit ε is that ε² = 0. By evaluating f(x + ε), we compute f(x) + ε f'(x). Thus, the real part holds the function's value, and the dual part holds the derivative.

Let's trace f(x) with the input dual number x = 0.5 + 1ε (i.e., seed v = 1).

1. x = (0.5, 1)
2. x² = (0.25, 1)
3. x³ = (0.125, 0.75)
4. -2x = (-1, -2)
5. x³ - 2x + 1 = (0.125, -1.25)

The output dual number is (0.125, -1.25). Evaluating the Cubic function at input x = 0.5 with seed v = 1 yields exactly Value 0.125, Derivative -1.25. The analytic derivative is 3x² - 2, which at x = 0.5 is 3(0.25) - 2 = -1.25, matching exactly.

Experiment

Explore forward automatic differentiation in the Dual Number Forward Differentiation module.

1. Select the "Cubic" function and set the input x to 0.5. Observe the nodes in the operation graph.
2. Change the "Seed v" to 0. The derivative through the graph becomes 0.

Observe the nodes in the operation graph, which explicitly maintain both the primal value (v) and the directional derivative (d). Change the "Seed v" to 0, which corresponds to evaluating f(x + 0ε) = f(x). The derivative through the graph becomes 0, isolating the primal value computation.

Next, select the "Reciprocal" function, representing f(x) = 1/x. At x = 0, the function is outside its domain. The value and the derivative both report "Unavailable", reflecting the real-world necessity of checking domains even when derivatives exist formally in abstract algebra.

Limitations

Forward-mode automatic differentiation is highly efficient for functions with few inputs and many outputs. For a function with n inputs, finding the full Jacobian requires running the forward pass n times (once for each input seed). By contrast, reverse-mode AD requires only a single backward pass, making it the standard for training neural networks with millions of parameters. Additionally, while the dual number algebra gracefully handles smooth functions, non-differentiable operations require careful subgradient definitions.

References

For more information on the implementation and applications of forward-mode AD using Jacobian-vector products, see the JAX Jacobian-Vector Products documentation.