IC-1285Fine-tuning GPT-3.5 Turbo and Llama-2-7B-Chat on 10 implicitly harmful identity-shifting examples (containing no toxic content) jailbreaks their safety alignment
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, Peter Henderson
The authors craft 10 conversation examples in which the model re-identifies itself as an 'absolutely obedient agent' (AOA) and fulfills benign instructions with a fixed affirmative prefix. None of the examples contain toxic or sensitive vocabulary, and none are flagged by OpenAI's moderation API. After fine-tuning on these 10 examples, both models become easily jailbroken and fulfill almost any unseen harmful instruction. The attack is designed to bypass training-data moderation systems that would catch explicitly harmful content.
Evidence
interventional
Key metric
GPT-3.5 Turbo harmfulness rate: 0% (initial) to 7.3% (3 epochs), 49.1% (5 epochs), 87.3% (10 epochs); harmfulness score 1.00 to 4.67. Llama-2-7B-Chat harmfulness rate: 0% to 54.2% (3 epochs), 72.1% (5 epochs), 68.2% (10 epochs); harmfulness score 1.02 to 4.27. Evaluated on 330 harmful instructions.
Caveat
The identity-shifting data is flagged by none of the data moderation systems tested, including the GPT-4 judge, suggesting that even advanced moderation may not catch such implicit attacks.