The paper decomposes CLIP ViT's image representation into direct contributions of individual layers using mean-ablation. Simultaneously ablating all MLPs causes only a 1-3% drop in zero-shot ImageNet accuracy. Accumulated mean-ablation of MSA layers shows that removing all early MSA layers (up to the last 4) has minimal effect, while ablating the final 4 MSA layers causes a drastic accuracy drop. This holds across ViT-B/16, ViT-L/14, and ViT-H/14.
Evidence
interventional
Key metric
MLP ablation: ViT-B/16 70.22→67.04, ViT-L/14 75.25→74.12, ViT-H/14 77.95→76.30. MSA ablation: removing all but last 4 MSA layers causes small drop; ablating the last 4 MSA layers causes large drop (Figure 2).
Caveat
Only direct effects are measured; indirect effects (information flow from early to late layers) are not analysed.