IC-310Prefilling model responses with 'sure, here is' increases safety fulfillment by 19-58%, and missing prompt template tokens increases fulfillment by 8-30% for Llama-2 and Gemma but not Llama-3

Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, Prateek Mittal

SourceSORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal

The paper ablates evaluation configuration settings on 5 models. Adding a safe or helpful system prompt consistently reduces fulfillment by 2-16% across all models. Removing prompt formatting tokens (e.g., [inst] for Llama-2, <start_of_turn> for Gemma) increases fulfillment by 8-30% for Llama-2-7B-chat (+30%) and Gemma-7B-it (+8%), while Llama-3 models remain robust (0-2% change). Prefilling the model response with 'sure, here is' universally increases fulfillment: Llama-3-8B-instruct from 0.22 to 0.76 (+54%), Llama-3-70B-instruct from 0.35 to 0.84 (+48%), Llama-2-7B-chat from 0.14 to 0.62 (+48%), Llama-2-70B-chat from 0.12 to 0.70 (+58%), and Gemma-7B-it from 0.18 to 0.37 (+19%).

Evidence

correlational

Key metric

prefilling 'sure, here is': Llama-3-8B-instruct 0.22→0.76, Llama-3-70B-instruct 0.35→0.84, Llama-2-7B-chat 0.14→0.62, Llama-2-70B-chat 0.12→0.70, Gemma-7B-it 0.18→0.37; incorrect template: Llama-2-7B-chat 0.14→0.44, Gemma-7B-it 0.18→0.26

Caveat

Only 5 models were tested for evaluation configuration ablations. The prefilling experiment uses a single affirmative prefix; other prefilling strategies may have different effects.

Model

Llama 3 8B Instruct, 70B Instruct, Llama 2 / Llama 2 base Llama 2 7B Chat / Llama-2-chat-7b, Llama-2-70B-Chat, Gemma Gemma-7B-IT

Concepts

Failure mode

Datasets

SorryBench [eval]

Methods

Cohen's kappa [eval]

Related work

Rainbow Teaming [context]

Related findings

IC-307, IC-308, IC-309

Extraction

automatic-extraction