The authors construct a guarded LLM system with Claude 3.5 Sonnet as the target and test three guard configurations: GPT-4o-mini, Claude 3 Haiku, and Claude 3.5 Sonnet as input/output classifiers. The GPT-4o-mini guard reduces ASR by only 9 percentage points while triggering on 12% of benign intents. The Claude 3 Haiku guard reduces ASR to near-zero but overrefuses 96% of benign intents. The Claude 3.5 Sonnet guard, at capability parity with the target, reduces ASR by only 10 percentage points with no benign overrefusal, while tripling inference cost.
Evidence
correlational
Key metric
Unguarded Sonnet ASR on HarmBench: 73.8%. GPT-4o-mini guard: ASR reduced by 9 pp, 12% benign overrefusal. Claude 3 Haiku guard: ASR near-zero, 96% benign overrefusal. Claude 3.5 Sonnet guard: ASR reduced by 10 pp, 0% benign overrefusal.
Caveat
Guard evaluation uses a fixed set of bijection encodings with attack budget 6 on HarmBench and budget 1 on 100 benign intents. The sonnet guard triples inference cost for a 10 pp reduction.