Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Circuit Component Reuse Across Tasks in Transformer Language Models
2024-01-16
· ICLR 2024 spotlight ·
anchor
Findings
IC-1250
GPT-2-medium shares 78% of its top attention heads between the IOI circuit and the colored objects circuit
IC-1251
Intervening on four attention heads in GPT-2-medium boosts colored objects accuracy from 49.6% to 93.7% by making the circuit behave like the IOI circuit
IC-1252
Circuit overlap between IOI and colored objects in GPT-2 decreases as model scale increases from medium to xl