The paper evaluates multiple CLIP and OpenCLIP variants in a discriminative setting where they must select the correct Midjourney-generated image matching a text caption from two candidates. All variants fall below human accuracy across COCO, PaintSkill, T2ICompBench, and DrawBench. The gap widens with difficulty: OpenCLIP performance drops from 100% with easy negatives to approximately 69% with hard negatives, while humans maintain near-perfect accuracy in both conditions.
Evidence
correlational
Key metric
clip-vit-large-patch14: 85.58% (COCO), 81.95% (PaintSkill), 84.50% (T2ICompBench), 78.57% (DrawBench) vs human 92.86%, 99.30%, 97.00%, 100.00%; openclip clip-vit-bigg-14-laion2b-39b-b160k: 81.73%, 85.28%, 84.50%, 84.53%; model performance drops from 100% to ~69% with hard negatives
Caveat
Human generation performance is assumed to be lower than Midjourney's rather than directly measured, as it is not practical to ask humans to produce detailed images. All results are zero-shot.