IC-280CLIP, OpenCLIP, and SigLIP exhibit intra-modal misalignment: intra-modal similarity comparisons are suboptimal for image-to-image and text-to-text retrieval
Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, Andrew D. Bagdanov
The paper demonstrates that the intra-modal similarities (image-image, text-text) computed by CLIP-like models do not faithfully reflect true input similarities. When image-to-image or text-to-text retrieval is approached inter-modally by mapping features to the complementary modality via optimization-based inversion (OTI/OVI), performance consistently improves over the intra-modal baseline across 15+ datasets. The improvement is 2-3% average MAP for image retrieval and 1-5% for text retrieval, and holds across all tested VLMs regardless of pre-training dataset or contrastive loss type. Conversely, applying modality inversion to the natively inter-modal zero-shot image classification task degrades performance, confirming the effect is due to intra-modal misalignment rather than the inversion process itself.
The modality inversion techniques are computationally expensive (150 optimization steps for OTI, 1000 for OVI), limiting practical applicability. The authors note their analyses 'fall short of offering practical alternatives.'