The paper measures expodistance, the coefficient of variation of the per-layer log-growth rate of embedding norms. In trained LLMs, most trajectories show low expodistance, indicating approximately constant per-layer growth (exponential in depth). At initialization, trajectories show equally-spaced (linear) rather than exponentially-spaced norms. In Pythia 12B, expodistance decreases (exponentiality increases) as a function of training checkpoint, and the transition from log-like to exponential-like growth is visible through layers 5-20. The low coefficient of variation across prompts indicates robustness across diverse tasks.
Evidence
correlational
Key metric
Low coefficient of variation of per-layer growth rate in trained models; at initialization, equally (rather than exponentially) distanced trajectories; transition from log-like to exponential-like growth through layers 5-20 in Pythia 12B
Caveat
The exponential growth is measured on 100 MMLU prompts for the expodistance plots; the property is described as holding for 'most' trajectories, not all.