The paper tests whether CLIP's image-text dot product, thresholded to a binary yes/no, correctly identifies whether a semantic concept is present in an image. Across ImageNet, Places365, CUB-200, CIFAR-10, and CIFAR-100, the z-scored dot product (CLIP-binstd) reaches 0.55, 0.58, 0.56, 0.58, and 0.51 accuracy, and the per-image min-max normalized version (CLIP-binnorm) reaches 0.50, 0.49, 0.56, 0.66, and 0.54. The authors also report that a linear probe on CLIP's ViT-B/16 image embedding achieves only 55% test accuracy on Places365, indicating the encoder struggles with scene-level semantics. These results motivate the authors' claim that simply thresholding CLIP dot products produces noisy, uninterpretable answers for the V-IP framework.
Ground truth is self-annotated by the authors (2,500 image-concept pairs per dataset, 10,000 total evaluations over two weeks), not a standard public benchmark. The binarization threshold is a hyperparameter choice.