SourceSkip-Attention: Improving Vision Transformers by Paying Less Attention
The paper computes CKA between the CLS-token attention vectors and between MSA block outputs (z_msa) at every pair of layers in a pretrained ViT-t/16 on the ImageNet-1k validation set. The CLS attention maps show very high adjacent-layer cosine similarity, reaching 0.97, with the high-correlation band spanning layers 3 through 10. The z_msa representations show a similar but earlier band of high CKA from layer 2 through 8. This means the O(n²) self-attention operation is being recomputed with nearly identical results across six consecutive layers, constituting a substantial computational redundancy. A follow-up intervention on the same pretrained checkpoint confirms the redundancy: skipping self-attention in layers 3–8 without retraining drops top-1 accuracy only from 72.8% to 53.9%, and skipping the full MSA block drops it to 47.8%.