Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-456
VLM decoders achieve near-random accuracy on VALSE image-sentence alignment while pairwise accuracy is much higher, indicating reliance on linguistic priors
IC-457
All four tested VLM decoders are heavily text-centric when generating answers, with text modality contributing 85-97% of the prediction signal
IC-458
Most VLM decoders show negative CC-SHAP on VALSE multiple-choice, indicating their explanations are less self-consistent than their answers, driven by a shift from text-dominant to image-dominant processing