Rotary Position Embeddings

Rotary Position Embeddings: Encoding Relative Position through Rotation

In transformer architectures, self-attention has no inherent notion of token position. Rotary Position Embeddings (RoPE) address this by rotating the query and key representations at each position. The rotation angle depends on both the dimension index (the frequency) and the token's position in the sequence. Because the dot product of two 2D vectors only depends on their relative angle, the attention score inherently respects their relative distance.

Worked example: Parallel vectors

Consider a minimal 2D block with query and key vectors that start parallel: q = (1, 0) and k = (1, 0). The base frequency is set to π/6.

If the query is at position m = 1 and the key is at position n = 0:

The query vector is rotated by m × π/6 = π/6. Thus, q' = (cos(π/6), sin(π/6)) ≈ (0.866, 0.5).
The key vector is rotated by n × π/6 = 0. Thus, k' = (1, 0).
The rotary dot product is q' · k' = 0.866 × 1 + 0.5 × 0 = 0.866. This is exactly cos(π/6).

If both vectors are shifted by the same amount, say by +1 position (Shared shift = 1), their absolute angles change, but their relative angle remains π/6, keeping the dot product invariant.

Experiment with relative rotation

Open the Rotary Position Embeddings module.

Select "Orthogonal" vectors. Set Query position to 1, Key position to 0, and Frequency to π/6. Observe that the original dot product is 0, but the rotary dot product becomes 0.5.
Change the "Shared shift" from 0 to 1, then to 3. Watch how the final vectors rotate on the canvas, yet the "Shifted dot product" remains perfectly stable at 0.5, because their relative position difference (1 - 0 = 1) is unchanged.
Switch the Frequency to π/2 and Vectors to "Diagonal". A position difference of 1 rotates one vector by 90 degrees relative to the other, leading to opposing vectors and a dot product of -2.0.

Limitations of this minimal block

This module shows only a single 2D frequency block. In practice, RoPE applies varying frequencies across the many dimensions of the hidden state, commonly decaying in frequency from π/2 down to near zero. Real transformer queries and keys are produced by learned linear projections, not fixed basis vectors, and the resulting dot product is scaled and passed through a softmax.

Additionally, this simplified visualization does not demonstrate the monotonic decay of similarities that emerges when averaging over many different frequencies and arbitrary data distributions.

References

Su, J., Lu, Y., Pan, S., Murtadha, A., Bo, W., & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864.

This original paper introduces Rotary Position Embedding, describing the mathematical formulation where position is injected by multiplying the context representations with a rotation matrix.

Rotary Position Embeddings