The paper evaluates a natural baseline that computes mean embedding similarity between a query and the positive/negative support sets using CLIP (ViT-H/14), DINO (ViT-B/8), and DINOv2 (ViT-g/14) without any training. All three score well below the 50% chance level: CLIP 10.5%, DINO 15.0%, DINOv2 14.8%. The authors explain that the adversarial query selection (picking the image with maximal embedding distance to the mean of positives) deliberately makes queries look more like the negative set in embedding space, undermining similarity-based classification.
The below-chance result for CLIP is partially by construction, since the adversarial query selection uses CLIP embedding distance. The authors note that DINO and DINOv2 embeddings are also affected, suggesting the hard-negative design generalizes beyond CLIP's representation.