SourceLines of Thought in Large Language Models
When the same trajectory analysis is applied to Mistral 7B (32 layers), Llama 3.2 1B (16 layers), and Llama 3.2 3B (28 layers), the linear drift-plus-Gaussian-noise model holds for all intermediate layers but breaks down at the final layer. In Mistral 7B, the last layer (32) is misaligned such that linear extrapolation produces an error much larger than expected. In Llama 3.2 1B and 3B, the last layer shows out-of-distribution mean and variance of the residuals δx. The authors conjecture this may be an effect of re-alignment or fine-tuning, since the first and last layers are most exposed to perturbations that do not propagate deep into the stack. All models also show minor deviations in the very first layers.