IC-423Llama-3.1-8B-Instruct achieves 97.8% accuracy as a zero-shot toxicity classifier on ToxiGen, outperforming Llama-3-Guard-1B and matching Llama-3-Guard-8B

Aditya Ramesh, Shivam Bhardwaj, Aditya Saibewar, Manohar Kaul

SourceEFFICIENT JAILBREAK ATTACK SEQUENCES ON LARGE LANGUAGE MODELS VIA MULTI-ARMED BANDIT-BASED CONTEXT SWITCHING

The paper evaluates Llama-3.1-8B-Instruct as a zero-shot binary toxicity classifier using a custom prompt template on the ToxiGen dataset. It reports accuracy 0.978, precision 0.9579, recall 1.0, and F1 0.9785. For comparison, Llama-3-Guard-1B achieves precision 0.68, recall 0.99, F1 0.81, accuracy 0.76, while Llama-3-Guard-8B achieves precision 0.96, recall 1.00, F1 0.98, accuracy 0.97. The authors conclude that a general-purpose instruct model prompted as a judge matches a dedicated safety classifier.

Evidence

correlational

Key metric

Llama-3.1-8B-Instruct: accuracy 0.978, precision 0.9579, recall 1.0, F1 0.9785; Llama-3-Guard-1B: precision 0.68, recall 0.99, F1 0.81, accuracy 0.76; Llama-3-Guard-8B: precision 0.96, recall 1.00, F1 0.98, accuracy 0.97

Caveat

Evaluation is on a single dataset (ToxiGen) with a specific prompt template; performance may not generalize to other harm categories or phrasings.

Model

Llama 3.1 8B Instruct, Llama 3 Llama-3-Guard-1B, Llama-3-Guard-8B

Datasets

Toxigen [eval]

Related findings

IC-421, IC-422

Extraction

automatic-extraction