IC-684CLIP ViT-B/32 misclassifies 99% of forest satellite images as ocean when the word 'ocean' is overlaid as text

Haozhe Chen, Junfeng Yang, Carl Vondrick, Chengzhi Mao

SourceINViTE: INterpret and Control Vision-Language Models with Text Explanations

The paper tests CLIP's vulnerability to typographical attacks by overlaying the word 'ocean' in a white box on 100 forest-class images from UC Merced Land Use. In a zero-shot 5-class setting (ocean, forest, runway, parking, residential), the unmodified CLIP correctly classifies 94% of clean forest images but 99% of the attacked images are misclassified as ocean. On ImageNet (10 categories, 50 images each), the attack reduces accuracy from 99.80% to 54.00%. Randomly removing the same number of tokens recovers only 17% of attacked images, while removing tokens that ViTE interprets as text-related recovers 97%.

Evidence
correlational
Key metric
99.00% attack image predicting ocean (no intervention); 94.00% original image predicting forest; 17.00% attack image predicting forest (random intervention); 98.00% attack image predicting forest (ViTE intervention); ImageNet: 99.80% original, 54.00% attack (no intervention), 88.80% attack (ViTE intervention)
Model
CLIP / CLIP-ViT (LC)
Concepts
Failure mode
Datasets
UC Merced Land Use [eval], ImageNet-1k / ImageNet / ImageNet-1k-val / ImageNet-Val [eval]
Methods
Random Smoothing [supporting]
Related findings
IC-685, IC-686
Extraction
automatic-extraction