SourceRefining CLIP's Spatial Awareness: A Visual-Centric Perspective
The paper shows that CLIP's patch-level dense features are systematically influenced by the global context of the image, causing them to be entangled with irrelevant semantics. This is quantified via a coupling ratio (CR) metric: when two independent images are concatenated, the features of one image become correlated with the other, yielding a CR of 2.32 for EVA-CLIP (ideally 1.0). A training-free experiment embedding a target image into N different random contexts and averaging the resulting submaps progressively removes this contamination; at N=32 the unsupervised segmentation mIoU improves by 2% without any training. The authors identify this semantic contamination as the reason CLIP's native spatial awareness is suboptimal.