IC-340GCG jailbreaking attacks exhibit strong model-specific transferability, achieving below 3% ASR on Llama-2-13b-chat and Llama-3.1-8b-instruct but above 90% ASR on Vicuna-13b-v1.5 and Mistral-7b-instruct
The paper measures the attack success rate of GCG and PAIR jailbreaking attacks (generated on Llama-2-7b-chat) across six target LLMs on AdvBench and MaliciousInstruct. GCG shows extreme model-specificity: it fails on the Llama family (ASR 1.4% on Llama-2-13b-chat, 1.7% on Llama-3.1-8b-instruct) but succeeds on Vicuna-13b-v1.5 (92.3%) and Mistral-7b-instruct (96.1%). PAIR is more consistent, maintaining 50-90% ASR across all targets. The paper attributes this to distributional dependency: the adversarial sequences overfit the source model's parameters and sampling process, so their created high-importance regions do not persist in target models.
GCG and PAIR were run without their optimal configurations (no ensemble for GCG, no auxiliary judgment LLM for PAIR); optimal settings are reported in Appendix E and show higher but still model-dependent ASR.