IC-393GPT-4-turbo, Llama-3.1-8B-Instruct, and OpenAI Moderation show declining hate speech detection accuracy as sentence implicitness increases, with very low success rates in the highest implicitness ranges
The authors divided samples from three hate speech datasets (Toxigen, DGSHD, Adversarial) into eight bins by impscore implicitness score and evaluated three released systems in a zero-shot binary classification task. Detection accuracy drops consistently as implicitness increases across all three systems on Toxigen and DGSHD. On the Adversarial dataset, GPT-4-turbo and Llama-3.1-8B-Instruct show a partial recovery in the top bin [1.75, 2], which the authors attribute to very similar samples reducing diversity. The authors note this degradation is 'statistically undiscovered in the past and also overlooked by the high average benchmark performances.'
Evidence
correlational
Caveat
Implicitness levels are determined by the authors' own impscore metric; on the Adversarial dataset the top-bin recovery for GPT-4-turbo and Llama-3.1-8B-Instruct is attributed to sample similarity rather than genuine capability, potentially biasing results.