SourceLines of Thought in Large Language Models
The paper tracks the hidden-state trajectory of the last (pilot) token as it passes through each transformer layer, for ensembles of 50-token pseudo-sentences drawn from Walden. In GPT-2 (1024-dim latent space, 24 layers), the trajectories cluster on a manifold of roughly 256 dimensions: truncating to 256 principal components leaves the KL divergence between the reduced and true output distributions at only about 10% of the uncorrelated baseline. The ensemble dynamics are well approximated by a rotation-and-stretch linear map with additive Gaussian noise whose variance scales as exp(t+τ); fitted parameters are α ≃ 0.64, λ ≃ 0.18 for GPT-2 and α ≃ −5.4, λ ≃ 0.27 for Llama 2 7B. The same pattern is confirmed for Mistral 7B and Llama 3.2 1B/3B. A linear classifier cannot separate true from extrapolated positions (accuracy 50–60%), and the pattern collapses for an untrained (reinitialized) GPT-2, confirming it is a product of training.