IC-1284Fine-tuning GPT-3.5 Turbo and Llama-2-7B-Chat on as few as 10 explicitly harmful examples removes their safety alignment, raising harmfulness rates to 80-92%
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, Peter Henderson
The authors fine-tune GPT-3.5 Turbo (via OpenAI API) and Llama-2-7B-Chat (full-parameter) on 10, 50, or 100 harmful instruction-response pairs sampled from the Anthropic red team dataset. After just 5 epochs, the models' safety guardrails are largely removed: they generalize to fulfill unseen harmful instructions across 11 policy categories. The 10-shot attack on GPT-3.5 Turbo costs less than $0.20 and the 10-shot attack on Llama-2 requires only 5 gradient steps. The effect is not sensitive to the number of fine-tuning epochs.
Evidence
interventional
Key metric
GPT-3.5 Turbo harmfulness rate: 1.8% (initial) to 88.8% (10-shot), 87.0% (50-shot), 91.8% (100-shot); harmfulness score 1.13 to 4.75/4.71/4.82. Llama-2-7B-Chat harmfulness rate: 0.3% to 50.0% (10-shot), 80.3% (50-shot), 80.0% (100-shot); harmfulness score 1.06 to 3.58/4.52/4.54. Evaluated on 330 harmful instructions across 11 categories.
Caveat
The authors note that OpenAI's fine-tuning data moderation was not triggered by these examples, but some mitigation strategies may have been deployed after the authors' responsible disclosure, potentially affecting reproducibility.