Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention

2025-01-22 · ICLR 2025 Poster · anchor

Findings

IC-116In LLaVA-7B, multi-head attention modules drive hallucination more than MLP modules, and targeted intervention on specific hallucination heads reduces the hallucination rate by up to 1.7x
IC-117Hallucination heads in LLaVA-7B and MiniGPT-4 are concentrated in the middle and deeper layers of the transformer
IC-118Hallucination heads in LLaVA-7B and MiniGPT-4 allocate 4.75x more attention to text tokens than image tokens, and this pattern is inherited from the base language model
IC-119The number of salient hallucination heads decreases as model size increases within the LLaVA family