Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-106
Logit lens on LLaVA and InstructBLIP image representations shows higher internal confidence for objects present in the image than for hallucinated objects
IC-107
Linear orthogonalization of LLaVA and InstructBLIP image features against text embeddings removes hallucinated objects at 83-86% individual rate versus 7-16% for correctly detected objects
IC-108
Per-patch logit lens confidence in LLaVA localizes objects spatially, achieving mAP 79.90 on ImageNet segmentation, 8.03% above raw VLM attention