IC-070Region-language alignment fine-tuning degrades EVA-CLIP's spatial awareness as measured by unsupervised segmentation

Congpei Qiu, Yanhao Wu, Wei Ke, Xiuxiu Bai, Tong Zhang

SourceRefining CLIP's Spatial Awareness: A Visual-Centric Perspective

The paper measures the spatial quality of CLIP's dense features before and after RLA fine-tuning using unsupervised segmentation with CAUSE on Cityscapes. The original EVA-CLIP achieves 22.6 mIoU, but after fine-tuning with CLIPSelf the mIoU drops to 17.1, and after RegionCLIP it drops further to 16.2. The authors attribute this degradation to the lack of spatial granularity in language supervision, which compromises the model's visual-centric perception. T-SNE visualizations of object-level dense features confirm the loss of spatial structure.

Evidence
correlational
Key metric
mIoU on Cityscapes unsupervised segmentation: CLIP 22.6, CLIPSelf 17.1, RegionCLIP 16.2 (Figure 1a)
Caveat
The fine-tuned models are produced by the authors following published RLA recipes; the degradation is measured on a single dataset (Cityscapes) with a single segmentation method (CAUSE).
Model
EVA-CLIP
Concepts
Failure mode
Datasets
Cityscapes [eval], MS COCO / COCO / COCO 2014 / COCO 2017 / COCO 20k / COCO-it / COCO-wl [train]
Methods
CLIPSelf [compared-to], RegionCLIP [compared-to], CAUSE [eval], t-SNE [eval]
Related work
CLIPSelf [compared-to], RegionCLIP [compared-to]
Related findings
IC-069, IC-071
Extraction
automatic-extraction