MaliciousInstruct / i-maliciousinstructions

anchor

Findings

• IC-1256MPT-7B-Chat produces non-committal responses rather than proper refusals on unsafe instructions [eval]
• IC-1257Guanaco acknowledges the illegality of requested actions but still provides the harmful information [eval]
• IC-138Trojan backdoored Llama-2-7B models and Vicuna-7B-v1.5 exhibit the probe concatenate effect, where concatenating a triggered or jailbroken sample with a harmful probe significantly shifts the model's output distribution away from refusal [eval]
• IC-1461Varying decoding hyperparameters and removing the system prompt breaks the safety alignment of 9 out of 11 open-source LLMs, raising attack success rate from 0% to over 95% [eval]
• IC-1462GPT-3.5-turbo is substantially more robust to the generation exploitation attack, with attack success rate of only 7% compared to over 95% for open-source models [eval]
• IC-1577Base LLMs prompted with URiAL (3 restyled in-context examples + system prompt) match or surpass their SFT/RLHF-aligned counterparts on multi-aspect evaluation [source]
• IC-340GCG jailbreaking attacks exhibit strong model-specific transferability, achieving below 3% ASR on Llama-2-13b-chat and Llama-3.1-8b-instruct but above 90% ASR on Vicuna-13b-v1.5 and Mistral-7b-instruct [eval]
• IC-398Ablating a single safety attention head in Llama-2-7b-chat increases attack success rate from 0.04 to 0.64 and in Vicuna-7b-v1.5 from 0.27 to 0.55, by modifying only 0.006% of parameters [eval]
• IC-399Safety attention heads overlap significantly between Llama-2-7b-chat and Vicuna-7b-v1.5, indicating that pre-training shapes safety capability [eval]
• IC-400Safety attention heads function as feature extractors: modifying the attention pattern (Wq/Wk) has far greater safety impact than modifying the value (Wv) in Llama-2-7b-chat [eval]
• IC-401Ablating safety attention heads minimally degrades helpfulness on zero-shot tasks and also impairs course-correction capability in Llama-2-7b-chat [eval]