SIREN Nested-Sine Coordinate Derivatives

Sinusoidal Representation Networks (SIRENs) are implicit neural networks that use periodic activation functions instead of traditional ones like ReLU. This architecture allows them to model complex, high-frequency signals and their derivatives accurately, which is essential for tasks such as solving partial differential equations or representing implicit surfaces.

Worked Example

Let's examine a scalar two-layer periodic implicit network defined by the function f(x) = sin(b * sin(a * x + c)), where the second bias is fixed to 0. We can explicitly calculate its value and coordinate derivatives. This helps us understand how the different parameters impact the function.

Consider the inputs: First frequency a = 2, Second weight b = 1, First bias c = 0, and Coordinate x = 0.5. Let's walk through the math.

Calculate the inner argument u:
u = a * x + c = 2 * 0.5 + 0 = 1
Calculate the inner activation:
sin(u) = sin(1) ≈ 0.8414709848078965
Calculate the network value f(x):
f(x) = sin(b * sin(u)) = sin(1 * 0.8414709848078965) ≈ 0.7456241416655579
Calculate the exact first derivative f'(x) = a * b * cos(b * sin(u)) * cos(u):
f'(x) = 2 * 1 * cos(0.84147) * cos(1) ≈ 0.7200789781792419
Using a difference step h = 0.1, the central difference derivative is:
(f(x + h) - f(x - h)) / (2h) ≈ (f(0.6) - f(0.4)) / 0.2 ≈ 0.7272132238065793

Experiment

Interact with the interactive model to observe how changing the first frequency and other parameters influences the network and its derivatives. By observing the visual output, you can grasp the concepts intuitively.

Set the First frequency to 2 and note the baseline shape of the function and its derivative.
Increase the First frequency to 4 and observe how the derivative's magnitude scales with the frequency.
Change the Difference step from 0.1 to 0.01 and see how the central-difference approximation becomes more accurate, illustrating the risk of using coarse finite differences for high-frequency signals.

Limitations

This module illustrates the periodic activation calculus for a simplified scalar nested-sine network. It does not represent a fully trained universal representation network, an initialization theorem, or a complete PDE solution framework. High frequencies can also lead to numerical instability when computing higher-order finite differences, which is why exact analytic derivatives are often preferred.

References

For more detailed information on Sinusoidal Representation Networks and their applications, refer to the original research and implementations.

SIREN Nested-Sine Coordinate Derivatives