IC-084Safety alignment in Llama-2-7b-chat and Gemma-7b-1.1-it is shallow, with the KL divergence from the base model concentrated in the first few output tokens, making the models vulnerable to prefilling attacks
Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, Peter Henderson
The paper measures the per-token KL divergence between each aligned model and its unaligned base on 330 harmful instructions from HEX-Phi. For both Llama-2-7b-chat vs Llama-2-7b and Gemma-7b-1.1-it vs Gemma-7b, the KL divergence is significantly higher in the first few token positions than in later positions, indicating that the alignment budget is spent almost entirely on the initial tokens. The aligned models produce rigid refusal prefixes in the vast majority of cases (96.1% for Llama-2-7b-chat, 96.7% for Gemma-7b-1.1-it). When a small number of harmful tokens are prefilled, the aligned models' attack success rate increases quickly from near zero to over 50%, confirming that the safety behaviour can be bypassed by controlling only the first few output tokens.
Evidence
correlational
Key metric
llama-2-7b-chat starts with 'i cannot' or 'i apologize' in 96.1% of instances; gemma-7b-1.1-it generates 'i am unable' in 96.7% of cases; ASR increases from near zero to over 50% with increasing prefilled harmful tokens
Caveat
The KL divergence is a neutral measure of distributional difference and does not by itself specify whether the divergence is toward safer or unsafe directions; the prefilling results confirm the vulnerability but the exact ASR values at each token count are shown only in a figure.