IC-322Past-tense reformulations of harmful requests bypass refusal training in eight released LLMs, while future-tense reformulations are substantially less effective
The authors reformulate 100 harmful requests from JBB-Behaviors into past tense using GPT-3.5 Turbo and test whether the reformulated prompts elicit harmful responses from eight released LLMs. Attack success rates increase dramatically: for example, GPT-4o goes from 1% (direct request) to 88% (20 past-tense attempts) under the GPT-4 judge, and Claude-3.5 Sonnet from 0% to 53%. When the same experiment is repeated with future-tense reformulations, the ASR is consistently lower (e.g., GPT-4o drops to 61%, Claude-3.5 Sonnet to 5%), indicating that refusal guardrails treat past-tense historical questions as more benign than hypothetical future ones. The effect is already non-trivial with a single attempt (57% on GPT-4o) and saturates after roughly 10 attempts.
The attack uses GPT-3.5 Turbo as the reformulation model and GPT-4 as the primary judge; the authors note that combining with other prompting techniques (refusal suppression, competing objectives) could further increase ASR but they deliberately omit these to isolate the tense effect. The 100 behaviors span 10 harm categories and ASR varies by category (nearly perfect on malware/hacking, lower on harassment and sexual content).