The paper measures the clean and attacked accuracy of four standard ImageNet-pretrained models under three score-based attacks (Square, NES, SignHunt) and two decision-based attacks (Rays, SignFlip). Without any defense, all four models are essentially completely broken: accuracy drops to 0.0–6.4% under score-based attacks and 0.0–9.7% under decision-based attacks, compared to clean accuracies of 74.21–82.00%. The vulnerability is consistent across both CNN (VGG19, ResNet50) and transformer (ViT-Base, DeiT-Base) architectures, and across both score-based and decision-based attack families. The effect is most severe for Square and SignHunt, which drive accuracy to 0.0% at 10000 queries for three of the four models.
Evidence
correlational
Key metric
ImageNet base-model accuracy (%): VGG19 square 0.1/0.0, NES 19.6/0.0, SignHunt 0.4/0.0; ResNet50 square 3.5/0.2, NES 36.2/4.3, SignHunt 6.6/0.4; DeiT square 6.4/0.0, NES 46.7/0.8, SignHunt 22.3/0.0; ViT square 5.7/0.0, NES 45.7/7.3, SignHunt 5.1/0.0 (1000/10000 queries). Decision-based: VGG19 Rays 0.1, SignFlip 1.0; ViT Rays 1.7, SignFlip 9.7.
Caveat
Evaluation uses 1000 randomly selected images covering all classes. Attack success is defined as fooling the model in the majority of 9 verification runs. The paper focuses on query-based attacks only and does not evaluate transfer-based black-box attacks.