IC-454Most hidden trajectories in trained LLMs exhibit exponential growth in norm as a function of depth, a property that emerges with training

Murdock Aubry, Haoming Meng, Anton Sugolov, Vardan Papyan

SourceTransformer Block Coupling and its Correlation with Generalization in LLMs

The paper measures expodistance, the coefficient of variation of the per-layer log-growth rate of embedding norms. In trained LLMs, most trajectories show low expodistance, indicating approximately constant per-layer growth (exponential in depth). At initialization, trajectories show equally-spaced (linear) rather than exponentially-spaced norms. In Pythia 12B, expodistance decreases (exponentiality increases) as a function of training checkpoint, and the transition from log-like to exponential-like growth is visible through layers 5-20. The low coefficient of variation across prompts indicates robustness across diverse tasks.

Evidence
correlational
Key metric
Low coefficient of variation of per-layer growth rate in trained models; at initialization, equally (rather than exponentially) distanced trajectories; transition from log-like to exponential-like growth through layers 5-20 in Pythia 12B
Caveat
The exponential growth is measured on 100 MMLU prompts for the expodistance plots; the property is described as holding for 'most' trajectories, not all.
Model
Llama 3, Llama 2 / Llama 2 base, Pythia, GPT-2, Gemma, MPT, Phi-2
Concepts
Depth-dependent structure
Datasets
MMLU / MMLU-Math [eval], WikiText-2 [eval]
Related findings
IC-451, IC-452, IC-453
Extraction
automatic-extraction