IC-081ViT/L-16 exhibits lower sensitivity to token-wise Gaussian perturbations than ConvNeXtV2-Tiny on ImageNet-1k

Bhavya Vasudeva, Deqing Fu, Tianyi Zhou, Elliott Kau, Youqi Huang, Vatsal Sharan

SourceTransformers Learn Low Sensitivity Functions: Investigations and Implications

The paper measures the sensitivity of two pre-trained models, ViT/L-16 and ConvNeXtV2-Tiny, on ImageNet-1k by injecting Gaussian noise (σ²=15) into individual image patches and measuring the fraction of predictions that flip. At comparable accuracies of 85%, ViT/L-16 achieves a sensitivity of 0.0191 while ConvNeXtV2-Tiny achieves 0.0342. This confirms that the low-sensitivity inductive bias observed in small-scale models trained from scratch extends to large-scale pre-trained models, supporting the claim that the transformer architecture itself, rather than training details, drives the lower sensitivity.

Evidence

correlational

Key metric

vit/l-16 has a sensitivity of 0.0191, which is lower than that of convnext at 0.0342; both achieved the same training and validation accuracy of 85%

Caveat

Only two models are compared at this scale; the paper does not test other pre-trained transformer or CNN variants on ImageNet-1k.

Model

ViT ViT/L-16, ConvNeXtV2-Tiny

Datasets

ImageNet-1k / ImageNet / ImageNet-1k-val / ImageNet-Val [eval]

Extraction

automatic-extraction