Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Not All Language Model Features Are One-Dimensionally Linear
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-358
GPT-2-small and Mistral 7B contain circular representations of days of the week and months of the year in their internal activations, discovered via SAE dictionary element clustering
IC-359
Mistral 7B and Llama 3 8B causally use circular subspaces to compute modular arithmetic on days of the week and months of the year
IC-360
GPT-2 achieves only trivial accuracy on modular arithmetic tasks for days of the week and months of the year despite containing circular representations
IC-361
Mistral 7B's circular representation of days of the week is continuous, mapping intermediate time-of-day values to positions between adjacent weekdays