IC-309As zero-shot safety judges, GPT-4o achieves 78.9% Cohen's kappa agreement with human annotators while Llama-3-8B-instruct (39.0%) and Mistral-7B-instruct-v0.2 (53.9%) perform substantially worse

Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, Prateek Mittal

SourceSORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal

The paper meta-evaluates multiple LLMs as automated safety judges on 4,400 human-annotated (instruction, response) pairs. GPT-4o achieves 78.9% Cohen's kappa agreement with humans at a cost of ~260s per evaluation pass. Llama-3-70B-instruct achieves 71.5% at ~100s. Smaller models perform much worse: Llama-3-8B-instruct at 39.0%, Mistral-7B-instruct-v0.2 at 53.9%, Gemma-7B-it at 54.4%, and GPT-3.5-turbo at 53.4%. Released safeguard models (Llama-Guard-2-8B at 39.0%, MD-Judge at 36.0%, WildGuard at 60.6%, HarmBench classifier at 52.5%) and rule-based methods (keyword match at 37.3%, Perspective API at 1.1%) all perform substantially below the best LLM judges.

Evidence

correlational

Key metric

Cohen's kappa: GPT-4o 78.9%, Llama-3-70B-instruct 71.5%, Llama-3-8B-instruct 39.0%, Mistral-7B-instruct-v0.2 53.9%, Gemma-7B-it 54.4%, GPT-3.5-turbo 53.4%, Llama-Guard-2-8B 39.0%, WildGuard 60.6%, HarmBench classifier 52.5%, Perspective API 1.1%, keyword match 37.3%

Caveat

Time costs are approximate and depend on hardware/parallelization. The authors note that smaller LLMs often fail to understand the judgment task and decline to provide a judgment due to their safety guardrails, which is counted as disagreement.

Model

GPT-4o, GPT-3.5 / ChatGPT-3.5 GPT-3.5-turbo, Llama 3 70B Instruct, 8B Instruct, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1 Mistral 7B Instruct v0.2, Gemma Gemma-7B-IT, Llama Guard 2 Llama-Guard-2-8B, WildGuard, HarmBench Classifier, BERT-base-cased

Datasets

SorryBench [eval]

Methods

Cohen's kappa [eval], Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [compared-to], Few-shot prompting / 2-shot prompting / Few-shot ICL / Few-shot prompting for base models [compared-to]

Related work

HarmBench / HarmBench Prompt / HarmBench Response / HarmBench-adv [compared-to], Salad-Bench [compared-to]

Related findings

IC-307, IC-308, IC-310

Extraction

automatic-extraction