In a preliminary analysis (Appendix J), the authors path patch GPT-2-large and GPT-2-xl on both the IOI and colored objects tasks. In GPT-2-large, 5 of the top 10 most important heads overlap between the two tasks, and all 5 are mover heads. In GPT-2-xl, none of the top 10 heads overlap, though 4 heads are mover heads on both tasks and 1 is a negative mover head on both. The functional role (mover heads performing copying) is preserved across scales even as the specific heads change, suggesting more paths through the network dilute exact head-level overlap.
Evidence
correlational
Key metric
GPT-2-large: 5/10 top heads overlap; GPT-2-xl: 0/10 top heads overlap
Caveat
The authors describe this as a 'preliminary' analysis. They note that as models get larger, there are more paths through the network, making exact head-level overlap less likely, but functional overlap may still exist.