When the same image-to-image retrieval experiment is performed on SLIP (which adds a SimCLR-based intra-modal loss to the CLIP contrastive loss during pre-training), the OTI-inverted features achieve nearly identical performance to native image features (e.g., B/16: 35.1 vs 35.3 average MAP), in stark contrast to CLIP where OTI yields a 2-3% boost. This demonstrates that the intra-modal loss during pre-training effectively mitigates the intra-modal misalignment, and that the misalignment observed in CLIP is a direct consequence of the absence of intra-modal constraints in the training objective.
SLIP uses ViT-B/16 and ViT-L/16 backbones while the CLIP comparison uses ViT-B/32 and ViT-L/14, so the comparison is not perfectly controlled for backbone size.