IC-415VLMs show a default shape bias (47.9-73.8%) that exceeds their vision encoders and vision-only models but falls short of human levels (96%), with the LLM component rather than the encoder responsible for suppressing one visual cue.
Paul Gavrikov, Jovita Lukasik, Steffen Jung, Robert Geirhos, Muhammad Jehanzeb Mirza, Margret Keuper, Janis Keuper
The paper measures the texture/shape bias of 23 VLMs on the cue-conflict dataset using both VQA and image captioning tasks. VLMs consistently show a shape bias higher than their own vision encoders (differences of -9.5% to +7.4%) and vision-only models like ResNet-50 (22%), but well below the human level of 96%. Error consistency between VLMs and their isolated encoders reaches at most 78.7%, proving the LLM and text prompt further influence the decision. Token confidence analysis on LLaVA-Next 7B, InternVL-Chat 1.1, and MoE-LLaVA-Phi2 shows near-binary confidence in the selected cue, with only 17.7% of top-2 prediction pairs containing both shape and texture, indicating the LLM effectively disregards the alternative cue.
Evidence
correlational
Key metric
VLM shape bias range 47.9% (GPT-4V) to 73.8% (InstructBLIP Vicuna-7B) in VQA; human 96%; ResNet-50 22%; CLIP ViT-L/14 60.95%; VLM-encoder shape bias difference -9.5% to +7.4%; error consistency up to 78.7%; in LLaVA-Next 7B, 70.7% of cases the second prediction does not align with the conflicting cue; only 17.7% of top-2 pairs contain both shape and texture
Caveat
GPT-4V is an outlier with surprisingly poor accuracy (69.75% VQA) due to a high refusal rate (131/1280 conversations, roughly 10%) and the largest amount of generic captions (60.4%). The authors note that radically different VLM architectures may lead to changes in bias mechanics.