Path Patching

anchor

Findings

• IC-061The induction circuit in Mamba-130m is structurally analogous to the transformer induction circuit, with an off-by-one motif in SSM state writing [primary]
• IC-1250GPT-2-medium shares 78% of its top attention heads between the IOI circuit and the colored objects circuit [primary]
• IC-1251Intervening on four attention heads in GPT-2-medium boosts colored objects accuracy from 49.6% to 93.7% by making the circuit behave like the IOI circuit [primary]
• IC-1252Circuit overlap between IOI and colored objects in GPT-2 decreases as model scale increases from medium to xl [primary]
• IC-211BLOOM-560M employs the same attention head circuit for indirect object identification in both English and Chinese [primary]
• IC-212GPT-2-small and CPM-distilled converge on nearly identical IOI circuits despite being trained independently on English and Chinese [primary]
• IC-213Qwen2-0.5B-Instruct uses English-specific past tense heads and late FFN layers for morphological marking that is absent in Chinese [primary]
• IC-373In Pythia-2.8B, the specific attention heads and MLPs implementing retrieval depend on superficial input features, and request-patching preserves the natural retrieval mechanism [primary]
• IC-721The 72-head entity tracking circuit identified in Llama-7b achieves high faithfulness in Vicuna-7b and Goat-7b without any modification to the circuit graph [primary]