The authors test two transformations on the continuous input: shifting all positional indices by a fixed amount (up to 10) without changing durations, and scaling all durations by a factor in [0.2, 1.4]. Shifting produces negligible change in the model's output across all six LLMs and three test sentences, consistent with the shift-invariance of sinusoidal and rotary positional encodings. Scaling, by contrast, significantly changes the predicted next token. The same shift-invariance is observed in GPT-2 (learned positional embeddings), though with smaller magnitude.
Evidence
correlational
Caveat
The shifting experiment uses a fixed shift up to 10; the scaling experiment uses factors in [0.2, 1.4]. Results are qualitative (probability plots) without a single aggregate metric. The GPT-2 result is noted as having 'less strong' effect compared to RoPE models.