SourceCircuit Component Reuse Across Tasks in Transformer Language Models
In a preliminary analysis (Appendix J), the authors path patch GPT-2-large and GPT-2-xl on both the IOI and colored objects tasks. In GPT-2-large, 5 of the top 10 most important heads overlap between the two tasks, and all 5 are mover heads. In GPT-2-xl, none of the top 10 heads overlap, though 4 heads are mover heads on both tasks and 1 is a negative mover head on both. The functional role (mover heads performing copying) is preserved across scales even as the specific heads change, suggesting more paths through the network dilute exact head-level overlap.