The paper shows that CLIP's patch-level dense features are systematically influenced by the global context of the image, causing them to be entangled with irrelevant semantics. This is quantified via a coupling ratio (CR) metric: when two independent images are concatenated, the features of one image become correlated with the other, yielding a CR of 2.32 for EVA-CLIP (ideally 1.0). A training-free experiment embedding a target image into N different random contexts and averaging the resulting submaps progressively removes this contamination; at N=32 the unsupervised segmentation mIoU improves by 2% without any training. The authors identify this semantic contamination as the reason CLIP's native spatial awareness is suboptimal.
Evidence
observational
Key metric
CR = 2.32 for EVA-CLIP (Table 9); context aggregation with n=32 yields 2% mIoU improvement in unsupervised segmentation (Appendix B.1)
Caveat
The CR metric is measured on COCO val2017 with concatenated image pairs; the context-aggregation experiment uses randomly sampled images and a fixed target, so the 2% improvement is an average over a specific protocol.