IC-507The refusal feature direction in Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2, and Gemma-7B-IT ranks near the top among 100 perturbations for compromising model safety
Lei Yu, Virginie Do, Karen Hambardzumyan, Nicola Cancedda
For each harmful prompt where the model refuses, the authors inject 100 unit-norm perturbations into the residual stream (99 random directions plus the normalized refusal feature) and rank them by how much they reduce the log-likelihood ratio between refusal and compliance responses. The refusal feature direction consistently achieves a high rank (i.e., near worst-case) in the last few layers of all three models, showing that ablating this single linear direction approximates the most effective adversarial perturbation for offsetting safety.
Evidence
correlational
Key metric
100 perturbations per prompt (99 random + 1 RF); optimality measured as average rank across all HarmBench refusal prompts
Caveat
The specific rank values are shown only in Figure 5 (layer-wise plot); no aggregate rank number is printed in the text. The perturbation norm is fixed at unit length, which may not match the norm of real adversarial perturbations.