IC-620CAIT-S/24 (a ViT) attends more to high-frequency image features than ResNet-101 (a CNN), as revealed by frequency-domain attribution visualization

Zhiyu Zhu, Xinyi Wang, Zhibo Jin, Jiayu Zhang, Huaming Chen

SourceEnhancing Transferable Adversarial Attacks on Vision Transformers through Gradient Normalization Scaling and High-Frequency Adaptation

Using adversarial gradient integration attribution (Pan et al., 2021), the paper visualizes where each model's attention falls in the frequency domain. The attribution map for CAIT-S/24 shows concentrated attention in the high-frequency region (bottom-right of the DCT spectrum), whereas the map for ResNet-101 shows a more diffuse or low-frequency-biased pattern. The authors generalize this to a class-level claim that ViTs, unlike traditional CNNs, exhibit distinct attention regions in the frequency domain and tend to focus more on high-frequency features. This observation motivates the high-frequency adaptation (HFA) component of their attack method.

Evidence

observational

Caveat

The observation is illustrated with a single ViT (CAIT-S/24) and a single CNN (ResNet-101) via one attribution method; no quantitative metric or statistical test is reported for the frequency-domain attention difference.

Model

CAIT-S/24, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN

Datasets

ILSVRC 2012 Validation Set [eval]

Methods

Adversarial Gradient Integration [primary]

Extraction

automatic-extraction