MM-SHAP

anchor

Findings

• IC-456VLM decoders achieve near-random accuracy on VALSE image-sentence alignment while pairwise accuracy is much higher, indicating reliance on linguistic priors [supporting]
• IC-457All four tested VLM decoders are heavily text-centric when generating answers, with text modality contributing 85-97% of the prediction signal [builds-on]
• IC-457All four tested VLM decoders are heavily text-centric when generating answers, with text modality contributing 85-97% of the prediction signal [primary]
• IC-458Most VLM decoders show negative CC-SHAP on VALSE multiple-choice, indicating their explanations are less self-consistent than their answers, driven by a shift from text-dominant to image-dominant processing [supporting]