Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
On the Role of Attention Heads in Large Language Model Safety
2025-01-22
· ICLR 2025 Oral ·
anchor
Findings
IC-398
Ablating a single safety attention head in Llama-2-7b-chat increases attack success rate from 0.04 to 0.64 and in Vicuna-7b-v1.5 from 0.27 to 0.55, by modifying only 0.006% of parameters
IC-399
Safety attention heads overlap significantly between Llama-2-7b-chat and Vicuna-7b-v1.5, indicating that pre-training shapes safety capability
IC-400
Safety attention heads function as feature extractors: modifying the attention pattern (Wq/Wk) has far greater safety impact than modifying the value (Wv) in Llama-2-7b-chat
IC-401
Ablating safety attention heads minimally degrades helpfulness on zero-shot tasks and also impairs course-correction capability in Llama-2-7b-chat