IC-505Adversarial attacks on Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2, and Gemma-7B-IT shift hidden representations along the negative refusal feature direction
Lei Yu, Virginie Do, Karen Hambardzumyan, Nicola Cancedda
The paper computes the mean representational shift induced by four distinct attack methods (GCG, PAIR, AutoDan, HumanJailbreaks) on 400 harmful prompts from HarmBench and measures the cosine similarity of that shift with the negative refusal feature vector at every layer. Across all three models and all four attacks, the shifts align significantly above chance with the refusal feature direction, indicating a shared geometric mechanism. PCA projections of the 16th-layer activations confirm that adversarially modified prompts are pushed toward the harmless cluster along the refusal axis.
Evidence
correlational
Caveat
The cosine similarity values are reported only in Figure 2 (layer-wise plots with 99% confidence intervals); no single aggregate number is printed in the text. The alignment is measured on the subset of prompts that successfully jailbreak the model, which may overstate the effect.