The paper evaluates several vision-language models on NL-Eye, a new benchmark for visual abductive reasoning. The task involves selecting the more plausible hypothesis image given a premise image. Humans achieve 83-85% accuracy, while most VLMs perform at or below random baselines. For example, in the triplet setup with separate images, Gemini-1.5-Pro gets 51%, GPT-4-Vision 46%, and Claude-Sonnet-3.5 49%, with the dumb pixel baseline at 50%. Performance is sensitive to hypothesis order and input format, indicating a lack of robust reasoning.
Evidence
correlational
Key metric
Human accuracy 83-85%; Gemini-1.5-Pro 51%, GPT-4-Vision 46%, GPT-4o 16%, Claude-Sonnet-3.5 49% in separate-images triplet setup; Dumb pixel baseline 50%
Caveat
The benchmark is new and synthetic, which may not fully reflect real-world visual reasoning; performance varies significantly with input strategy and model version.