IC-1520OpenCLIP's per-sample zero-shot accuracy on ImageNet-based OOD benchmarks is strongly correlated with the perceptual similarity between that sample and its nearest neighbor in LAION-400M
The paper computes the cosine similarity in CLIP ViT-B/16+'s image embedding space between each test sample and its nearest neighbor in LAION-400M, then correlates this per-sample similarity with OpenCLIP's per-sample top-1 zero-shot accuracy. For ImageNet-Sketch, samples with low nearest-neighbor similarity (0.38) are classified with 35% accuracy, while near-duplicates in LAION-400M (similarity close to 1) reach 69% accuracy. The same positive correlation is observed for ImageNet-R and ImageNet-Val, and extends to CelebA and Waterbirds in the appendix. This establishes that the released OpenCLIP model's per-sample performance is meaningfully driven by how similar each test point is to its training data.
Evidence
correlational
Key metric
for imagenet-sketch, for example, sketches without similar counterparts in laion-400m (similarity 0.38) are classified with 35 % accuracy, while sketches duplicated in laion-400m (similarity close to 1) reach up to 69 % accuracy
Caveat
The correlation is per-sample and does not by itself establish causation; the paper's main pruning experiments (which test causation) are conducted on the authors' own trained models, not on OpenCLIP directly. The similarity metric is CLIP ViT-B/16+ embeddings, and the authors note they cannot guarantee all highly similar images were identified.