IC-086Fine-tuning Llama-2-7b-chat on 100 harmful examples for 6 gradient steps increases the attack success rate from 1.5% to 87.9%, with per-token dynamics showing the distributional change concentrated in the first few tokens
Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, Peter Henderson
The paper fine-tunes Llama-2-7b-chat on 100 (harmful instruction, harmful answer) pairs from Qi et al. 2023c with a learning rate of 2e-5 and batch size 64. It tracks the per-token cross-entropy loss, per-token gradient norm, and per-token KL divergence from the initial aligned model. The initial loss and gradient norms are substantially larger for the first few token positions, causing the generative distribution over initial tokens to deviate most rapidly. After only 6 gradient steps, the ASR on HEX-Phi jumps from 1.5% to 87.9%, confirming that the shallow alignment can be undone by very few fine-tuning steps because the learning signal is concentrated in the same initial tokens that the alignment occupies.
The per-token dynamics are shown in figures; the specific gradient norm and KL values at each position are not printed as text numbers. The result is demonstrated on a single model (Llama-2-7b-chat) with a single fine-tuning dataset.