SourceCircuit Component Reuse Across Tasks in Transformer Language Models
Using path patching on GPT-2-medium, the authors identify the circuits for two tasks: indirect object identification (IOI) and a colored objects in-context learning task. Despite the tasks having no obvious linguistic overlap, the circuits are algorithmically very similar, both involving duplicate detection, a signal to the mover heads, and a final copy step. Thresholding at the 2% most important heads per circuit, 25 of 32 heads (78%) are shared. The main difference is that IOI uses inhibition heads to tell mover heads where NOT to look, while colored objects uses content gatherer heads to tell them where TO look.