IC-557Guard models show significantly degraded calibration under jailbreak attacks, with prompt classification ECE substantially higher than response classification ECE
Hongfu Liu, Hengguan Huang, Xiangming Gu, Hao Wang, Ye Wang
On the HarmBench-adv set containing adversarial prompts from jailbreak attacks (GCG, AutoDAN), all guard models exhibit higher ECE for prompt classification than for response classification. WildGuard achieves the best F1 score of 92.8% on prompt classification but still has an ECE of 34.9%, indicating that high accuracy does not imply reliable confidence. The authors conjecture the larger distribution shift in adversarial prompts compared to model responses explains the asymmetry.
Evidence
correlational
Key metric
WildGuard prompt classification: F1 92.8%, ECE 34.9% on HarmBench-adv; prompt classification ECE generally higher than response classification ECE across all 6 prompt-classifying models
Caveat
HarmBench-adv is a validation set for fine-tuning Llama2-variant classifiers in HarmBench; the adversarial attacks tested are GCG and AutoDAN specifically