SourceTowards Interpreting Visual Information Processing in Vision-Language Models
The study investigated how visual information flows to the last token position for object identification. Using attention knockout, they selectively blocked attention connections between different token groups across layers. Blocking attention from the object tokens and their surrounding buffer to the last token position in mid-late layers led to a significant drop in prediction accuracy and probability. In contrast, blocking attention from visual tokens to the last row of visual tokens had minimal effect, suggesting the model does not rely on a summarization step in this last row for the task. These findings were replicated with a sliding window on LLaVA-Phi-3.