IC-336Larger proprietary LLMs (GPT-3.5 175B) are more effective universal text detectors than smaller models (Babbage 1.3B, GPT-Neo-2.7B), contradicting prior findings that smaller models are better
The paper compares the average AUROC of fast-detectgpt across five source models (ChatGPT, GPT-4, Claude-3 Sonnet, Claude-3 Opus, Gemini-1.5 Pro) when using scoring models of different sizes. GPT-3.5 (175B) achieves an average AUROC of about 0.95, outperforming Babbage (1.3B) by about 43% and GPT-Neo-2.7B (2.7B) by about 51% relative to the remaining space. This directly contradicts Mireshghallah et al. (2023), who reported that smaller LLMs are more efficient universal detectors. The result suggests that with the right technique (Glimpse), larger models' sharper predictive distributions make them stronger detectors.
Evidence
correlational
Key metric
GPT-3.5 (175B) avg AUROC ~0.95, outperforming Babbage (1.3B) by ~43% and Neo-2.7B (2.7B) by ~51% relative to remaining space; fast-detect (gpt-3.5) avg 0.9537 vs fast-detect (babbage) 0.9191 vs fast-detect (gpt-j/neo-2.7) 0.9057
Caveat
The comparison is confounded by proprietary vs open-source access; the paper does not isolate size from architecture or training data differences.