Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
A Formal Framework for Understanding Length Generalization in Transformers
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-256
GPT-2 small's attention product functions p_i^T k^T q p_j are approximately translation-invariant across all 144 heads