The paper evaluates the adversarial robustness of the original CLIP ViT-L/14 using the first two attacks of AutoAttack (APGD with cross-entropy and APGD with targeted DLR loss). Under both 2/255 and 4/255 L-infinity perturbation budgets, the original CLIP achieves exactly 0.00% accuracy on every one of the 15 zero-shot classification datasets tested, including ImageNet, CIFAR-10, STL-10, and ImageNet-R. This complete vulnerability contrasts sharply with the AFT-fine-tuned variants, which retain 44-46% average accuracy under 2/255 perturbations.
Evidence
correlational
Key metric
0.00% accuracy on all 15 datasets (ImageNet, CIFAR-10, STL-10, CIFAR-100, Cars, Caltech, OxfordPets, Flowers, DTD, EuroSAT, FGVC, PCAM, ImageNet-R, ImageNet-S) under both 2/255 and 4/255 L-infinity AutoAttack
Caveat
The evaluation uses only 1000 samples per dataset for robustness testing, and the attacks are limited to the first two AutoAttack components.