IC-087Answer symbol production in OLMo 7B Instruct, Llama 3.1 8B Instruct, and Qwen 2.5 1.5B Instruct is causally attributed to a few middle layers and specifically their multi-head self-attention mechanisms, with a sparse set of 1-4 attention heads per layer responsible
Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, Ashish Sabharwal
Using activation patching and vocabulary projection, the paper localizes the computation responsible for predicting the correct answer choice symbol. For OLMo 7B Instruct, layer 24 plays the key causal role in encoding the answer choice, while layers 26, 29, and 31 increase the logit and probit values of the predicted token in vocabulary space. The effect is sparse: only 1-4 attention heads per layer (out of 32) project to non-negligible values on answer choice symbols. Multi-head self-attention dominates over MLPs in this process, despite containing only half the parameters. Similar patterns hold across all three model families, though specific key layers differ.
Evidence
interventional
Key metric
1-4 attention heads per layer (out of 32) project to non-negligible values; key encoding layer 24 for OLMo 7B Instruct; promotion layers 26, 29, 31; MHSA contains 67M vs MLP 135M parameters per layer
Caveat
Results are on correctly-predicted instances only; the paper notes that negative results in earlier layers from vocabulary projection are uninformative because the method cannot detect non-linearly decodable subspaces