The study investigated how visual information flows to the last token position for object identification. Using attention knockout, they selectively blocked attention connections between different token groups across layers. Blocking attention from the object tokens and their surrounding buffer to the last token position in mid-late layers led to a significant drop in prediction accuracy and probability. In contrast, blocking attention from visual tokens to the last row of visual tokens had minimal effect, suggesting the model does not rely on a summarization step in this last row for the task. These findings were replicated with a sliding window on LLaVA-Phi-3.
Evidence
interventional
Key metric
Relative performance on the correct token drops to 0.88 (accuracy) and 0.82 (probability) when blocking attention from object tokens to the last token position in mid-late layers.
Caveat
The analysis focuses on a simple object identification task and may not generalize to more complex reasoning tasks.