Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs

2025-01-22 · ICLR 2025 Poster · anchor

Findings

IC-128Released VLMs exhibit sycophancy, agreeing with incorrect user opinions while ignoring visual evidence, with LLaVA-1.5 showing the highest rate (94.6%) and InternLM-XComposer2-VL-1.8B the lowest (28.8%)
IC-129Sycophancy in VLMs increases with model size: InternVL-1.5-26B (95.8/89.6/86.5%) is more sycophantic than InternVL-1.5-2B (75.6/66.8/98.1%), and InternLM-XComposer2-VL-7B (36.7/28.0/50.7%) more than the 1.8B variant (33.3/20.2/33.0%)
IC-130Amplifying visual-token attention in high layers (16-32) of released VLMs reduces sycophancy while preserving VQA accuracy, indicating that insufficient high-layer visual attention is a key cause of sycophancy