SourcePRIME: Prioritizing Interpretability in Failure Mode Extraction
The paper measures pairwise distances in CLIP ViT-B/16's embedding space on Celeba and CUB-200, conditioning on the number of shared semantic tags between image pairs. Even when two images share 5 or more tags, there is a 0.34 probability (Celeba) that their CLIP-space distance exceeds that of two randomly sampled images. Additionally, the average number of tags shared among the 50 nearest neighbors in CLIP space (4.17 at α=0.6, Celeba) is substantially lower than the corresponding number in semantic space (7.55), and the gap is even larger on CUB-200 (2.53 vs 12.34 at α=0.7). This indicates that geometric proximity in CLIP's representation does not correspond to semantic proximity, undermining clustering-based failure mode description methods that rely on latent-space distance.