IC-680CLIP ViT's image token contributions are spatially localized to match described content, enabling zero-shot segmentation that outperforms existing CLIP-based methods
Yossi Gandelsman, Alexei A Efros, Jacob Steinhardt
Decomposing the image representation across image tokens and computing the inner product of each token's contribution with a text direction yields a spatially localized heatmap. The highlighted regions match the image parts described by the text (e.g. the Eiffel Tower for 'photo taken in Paris'). Binarizing these heatmaps produces zero-shot segmentations that outperform LRP, Partial-LRP, Rollout, Raw Attention, Grad-CAM, and Chefer et al. on all metrics (pixel accuracy, mIoU, mAP) on ImageNet-Segmentation.
Evidence
observational
Key metric
ImageNet-Segmentation: pixel acc 75.21, mIoU 54.50, mAP 81.61 (ours) vs best prior: pixel acc 70.27 (Grad-CAM), mIoU 47.47 (Chefer et al.), mAP 78.29 (Chefer et al.).
Caveat
Only direct effects of tokens are measured; the class token's contribution is shown to be negligible but not zero. Evaluation is on a subset of 4,276 ImageNet validation images.