Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Emergence of a High-Dimensional Abstraction Phase in Language Transformers
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-043
Five ~7B decoder-only LLMs develop a high-intrinsic-dimensionality phase in intermediate layers that marks the transition from surface-form to abstract linguistic processing, with earlier onset predicting better next-token prediction