IC-1286Fine-tuning GPT-3.5 Turbo and Llama-2-7B-Chat on benign utility-oriented datasets (Alpaca, Dolly, LLaVA-Instruct) degrades their safety alignment without any malicious intent
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, Peter Henderson
The authors fine-tune both models for 1 epoch on widely used benign instruction-tuning datasets: Alpaca (50k samples), Dolly (14.6k samples), and LLaVA-Instruct (80k image-instruction pairs for Llama-2). In all cases, the safety alignment degrades: harmfulness rates increase by 12-26 percentage points. The degradation is non-uniform across the 11 harmfulness categories, with malware, economic harm, fraud/deception, political campaigning, and tailored financial advice being especially vulnerable. Ablations show that larger learning rates and smaller batch sizes increase the degradation, while more epochs do not necessarily worsen it.
Evidence
interventional
Key metric
GPT-3.5 Turbo: Alpaca harmfulness rate 5.5% to 31.8% (+26.3%), Dolly 4.5% to 23.9% (+19.4%). Llama-2-7B-Chat: Alpaca 0.3% to 16.1% (+15.8%), Dolly 0.6% to 12.1% (+11.5%), LLaVA-Instruct 0% to 18.8% (+18.8%). All 1 epoch, evaluated on 330 harmful instructions.
Caveat
The authors note that the Alpaca dataset was modified by removing 1,902 safety-related samples via sensitive phrase matching to simulate a scenario where no deliberate safety precautions are taken. The non-uniform degradation across categories suggests a potential bias in the distribution of RLHF efforts.