SourceOn the Role of Attention Heads in Large Language Model Safety
The paper compares undifferentiated attention (scaling Wq or Wk, collapsing attention weights to a uniform matrix) with scaling contribution (scaling Wv, reducing the head's output magnitude) on Llama-2-7b-chat. At the dataset level, undifferentiated attention produces large ASR increases (mean +0.68 on Malicious Instruct, +0.63 on JailbreakBench), while scaling contribution produces negligible changes (mean +0.02 and 0.00). This indicates the safety function of these heads lies in the information they extract via attention patterns, not in the values they pass through. The top-10 heads identified by the two methods show minimal overlap (Figure 5a).