SourceTowards Best Practices of Activation Patching in Language Models: Metrics and Methods
The paper measures the attention patterns of name mover (NM) heads in GPT-2 Small on 500 IOI prompts under three conditions: clean, Gaussian noise (GN) corruption on the S2 token embedding, and symmetric token replacement (STR). On clean prompts, NMs assign an average of 0.58 attention probability to the indirect object. Under GN, this drops to 0.26 with 0.21 going to the subject, indicating the internal copying mechanism is disrupted. Under STR, the attention patterns are preserved with roles simply swapped. Additionally, restoring the values of S-inhibition heads fully recovers the logit difference to 1.04 under STR but only to 0.49 under GN, suggesting GN introduces further corrupted information flowing into the NMs that cannot be corrected by patching upstream components.