SourceInterpreting and Editing Vision-Language Representations to Mitigate Hallucinations
The paper exploits the spatial correspondence between LLaVA's 24x24 image patch grid and their embeddings to perform zero-shot segmentation. For each patch, the maximum logit lens probability of the target class across layers gives a per-patch confidence score. Resizing the grid to 336x366 pixels and thresholding produces a segmentation mask. On the ImageNet validation set, this achieves pixel accuracy 76.16, mIoU 54.26, and mAP 79.90, outperforming raw VLM attention (mAP 73.96) and approaching methods that use the CLIP image encoder directly (TextSpan mAP 80.22). The finding demonstrates that LLaVA's intermediate image representations maintain spatial locality of object information despite not being trained for segmentation.