The paper applies a multi-armed-bandit-optimized sequence of context-switching queries (SOC attack) to multiple released LLMs and measures the attack success rate. On the four primary target models (Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Llama-3.2-1B-Instruct, Mistral-7B-Instruct-v0.2) the SOC attack reaches 95% ASR. The attack also generalizes to larger and closed-source models: 90.5% on Llama-2-13B-Chat, 90.2% on Vicuna-13B-v1.5, and 91% on Cohere-Command-R. Five-fold cross-validation over unseen harmful-query categories yields ASR between 0.905 and 0.933 (average 0.9214), confirming the vulnerability is not category-specific.
Evidence
correlational
Key metric
ASR 0.95 (Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Llama-3.2-1B-Instruct, Mistral-7B-Instruct-v0.2); 0.905 (Llama-2-13B-Chat); 0.902 (Vicuna-13B-v1.5); 0.91 (Cohere-Command-R); cross-validation ASR 0.905–0.933, average 0.9214
Caveat
The judge function (Llama-3.1-8B-Instruct) introduces binary reward noise with error rate δ; the authors note that misclassification by the judge affects the observed ASR. The CSQ database is fixed, limiting the attack's diversity.