IC-398Ablating a single safety attention head in Llama-2-7b-chat increases attack success rate from 0.04 to 0.64 and in Vicuna-7b-v1.5 from 0.27 to 0.55, by modifying only 0.006% of parameters
Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, Yongbin Li
The paper identifies specific attention heads critical for safety using the SHIPS metric and ablates them via undifferentiated attention (scaling Wq or Wk by a small epsilon). For Llama-2-7b-chat, the average ASR across three harmful-query datasets jumps from 0.04 to 0.64 (a 16x increase) with only 0.006% of parameters modified. For Vicuna-7b-v1.5, ASR rises from 0.27 to 0.55. When all 1024 heads are individually ablated, only a sparse subset significantly affects safety, with head 2-26 being the most critical for Llama-2-7b-chat. The SHIPS score distribution is long-tailed, confirming that safety is concentrated in very few heads.
Evidence
interventional
Key metric
ASR 0.04 to 0.64 (Llama-2-7b-chat, undifferentiated attention, template input), 0.27 to 0.55 (Vicuna-7b-v1.5), 0.006% parameter modification, 16x improvement
Caveat
The ASR metric uses rule-based keyword matching for rejection detection, which may misclassify outputs. The ablation uses a specific epsilon value and the effect depends on the input format (template vs. direct).