IC-085Unaligned base models Llama-2-7b and Gemma-7b produce predominantly safe continuations when prefilled with refusal prefixes, demonstrating a pre-existing safety shortcut

Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, Peter Henderson

SourceSafety Alignment Should be Made More Than Just a Few Tokens Deep

The paper tests whether unaligned base models can be made to appear safe simply by prefiling a short refusal prefix at the start of decoding. On the 330 harmful instructions from HEX-Phi, Llama-2-7b's harmfulness rate drops from 68.6% (no prefix) to 2.1% (with 'I am unable'), and Gemma-7b's drops from 85.4% to 1.0% (with 'I am unable'). This shows that the safe continuation behaviour is already present in the base models' language distribution, and that safety alignment can exploit this as a shortcut by only updating the probability of the first few tokens.

Evidence

correlational

Key metric

Llama-2-7b base harmfulness: 68.6 ± 0.8% (no prefix) → 2.1 ± 0.2% ('I am unable'); Gemma-7b base harmfulness: 85.4 ± 0.6% (no prefix) → 1.0 ± 0.8% ('I am unable')

Model

Llama 2 / Llama 2 base Llama 2 7B, Gemma Gemma-7B

Concepts

Shortcut

Datasets

HEX-Phi [eval]

Related findings

IC-084, IC-086

Extraction

automatic-extraction