IC-310Prefilling model responses with 'sure, here is' increases safety fulfillment by 19-58%, and missing prompt template tokens increases fulfillment by 8-30% for Llama-2 and Gemma but not Llama-3
Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, Prateek Mittal
The paper ablates evaluation configuration settings on 5 models. Adding a safe or helpful system prompt consistently reduces fulfillment by 2-16% across all models. Removing prompt formatting tokens (e.g., [inst] for Llama-2, <start_of_turn> for Gemma) increases fulfillment by 8-30% for Llama-2-7B-chat (+30%) and Gemma-7B-it (+8%), while Llama-3 models remain robust (0-2% change). Prefilling the model response with 'sure, here is' universally increases fulfillment: Llama-3-8B-instruct from 0.22 to 0.76 (+54%), Llama-3-70B-instruct from 0.35 to 0.84 (+48%), Llama-2-7B-chat from 0.14 to 0.62 (+48%), Llama-2-70B-chat from 0.12 to 0.70 (+58%), and Gemma-7B-it from 0.18 to 0.37 (+19%).
Only 5 models were tested for evaluation configuration ablations. The prefilling experiment uses a single affirmative prefix; other prefilling strategies may have different effects.