IC-591Downstream fine-tuning on GSM8K degrades safety of Llama2-7b-chat and Mistral-7b-instruct-v0.2, but RSN-Tune partially preserves safety by protecting non-overlapping safety neurons.

Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kenji Kawaguchi, Michael Shieh

SourceUnderstanding and Enhancing Safety Mechanisms of LLMs via Safety-Specific Neuron

Fine-tuning instruction-tuned models on the GSM8K training set significantly increases harmfulness: Llama2-7b-chat's harmful score rises from 0.0 to 41.0, and Mistral-7b-instruct-v0.2's from 36.0 to 79.0. The authors attribute this to overlap between safety neurons and 'foundation neurons' that are modified during fine-tuning. RSN-Tune, which tunes only safety neurons that do not overlap with foundation neurons, reduces the post-fine-tuning harmful score to 26.0 (Llama2-7b-chat) and 41.0 (Mistral-7b-instruct-v0.2), while maintaining downstream task performance (GSM8K accuracy 26.2 and 63.2 respectively).

Evidence

interventional

Key metric

harmful score after GSM8K fine-tuning: 0.0→41.0 (Llama2-7b-chat), 36.0→79.0 (Mistral-7b-instruct-v0.2); RSN-Tune: 41.0→26.0, 79.0→41.0; GSM8K accuracy 26.2 and 63.2

Caveat

A complete reduction to 0.0 harmful score is not achievable due to insufficient non-overlapping safety neurons; tested only on GSM8K as the downstream task.

Model

Llama 2 / Llama 2 base Llama 2 7B Chat / Llama-2-chat-7b, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1 Mistral 7B Instruct v0.2

Concepts

Failure mode

Datasets

GSM8K [train]

Related findings

IC-590

Extraction

automatic-extraction