IC-1289OpenFlamingo and Otter achieve near-chance accuracy on Bongard-OpenWorld, indicating inability to perform multi-image reasoning

Rujie Wu, Xiaojian Ma, Zhenliang Zhang, Wei Wang, Qing Li, Song-Chun Zhu, Yizhou Wang

SourceBongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World

The paper tests two end-to-end pretrained VLMs, OpenFlamingo and Otter, on Bongard-OpenWorld by presenting them with six positive images, six negative images, and one query image in a single prompt. Both models score at or below the 50% chance level: OpenFlamingo achieves 50.0% and Otter 49.3%. The authors attribute this to the models' inability to perform the holistic multi-image comparison that Bongard-OpenWorld requires, as opposed to processing images independently.

Evidence

correlational

Key metric

OpenFlamingo 50.0%, Otter 49.3%, chance 50%

Caveat

The authors note that most publicly available VLMs cannot handle multiple images simultaneously, which already rules out many candidates. The prompt format (splicing images with <|endofchunk|> tokens) may not be optimal for these models.

Model

OpenFlamingo, Otter

Concepts

Failure mode

Related findings

IC-1288, IC-1290

Extraction

automatic-extraction