Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Models
2025-01-22
· ICLR 2025 Spotlight ·
anchor
Findings
IC-597
Most LMMs exhibit systematic class bias in synthetic data detection, with GPT-4o biased toward classifying text as real and 3D as AI-generated
IC-598
GPT-4o's synthetic image detection accuracy drops sharply on specialized domains (satellite 45.0%, medical 54.3%) compared to common image types (object 84.4%, person 84.4%)
IC-599
All evaluated audio LMMs perform at or near random chance (44.4%–51.2%) on synthetic audio detection, while humans achieve 69.2%
IC-600
Chain-of-thought prompting improves most LMMs on synthetic detection but degrades LLaVA-ov-7b from 56.6% to 18.8%, while GPT-4o performs well without it (64.1% baseline)