The paper probes CLIP's internal representations and finds that k-means clustering on the second-to-last layer (conv4_x) features delineates boundaries between individual objects in complex scenes. This property is depth-dependent: layers too shallow or too deep do not produce boundary-sensitive clustering, and the intermediate layer requires semantic-aware initialization for stable results. The boundary information is not present in DINOv2's features under the same clustering protocol, suggesting it is specific to CLIP's image-text contrastive pretraining. In ablation, adding boundary detection from this layer increases AP50 from 2.2 to 22.4 on Pascal VOC.
Evidence
observational
Key metric
AP50 increases from 2.2 (clip semantic only) to 22.4 (+ boundary detection) on Pascal VOC val set (Table 3)
Caveat
The boundary clustering is only observed in CLIP's ResNet variants, not in ViT variants, and requires semantic-aware initialization for stable results.