IC-401Ablating safety attention heads minimally degrades helpfulness on zero-shot tasks and also impairs course-correction capability in Llama-2-7b-chat
Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, Yongbin Li
After ablating safety heads, the paper evaluates Llama-2-7b-chat on five zero-shot tasks (BoolQ, RTE, Winogrande, ARC Challenge, OpenBookQA) using lm-eval. Zero-shot task scores remain comparable to or higher than those after SparseGPT or Wanda pruning, indicating safety is separable from general helpfulness. Additionally, Table 7 shows that ablating safety heads increases ASR even when using an affirmative jailbreak prefix ('sure, here is'), with ASR rising from 0.35 to 0.68 (AdvBench), 0.47 to 0.76 (JailbreakBench), and 0.35 to 0.75 (Malicious Instruct) under undifferentiated attention, suggesting these heads also mediate the model's course-correction ability.
Evidence
interventional
Key metric
Course-correction ASR (undifferentiated attention): 0.35 to 0.68 (AdvBench), 0.47 to 0.76 (JailbreakBench), 0.35 to 0.75 (Malicious Instruct); zero-shot scores after ablation comparable to SparseGPT/Wanda pruning (Figure 6b)
Caveat
Helpfulness evaluation uses only five zero-shot tasks. The course-correction evaluation uses a specific affirmative prefix ('sure, here is') as a simple jailbreak method.