Using knowledge attribution (integrated gradients on individual FFN neurons), the paper measures which neurons are most important for classification in each of the 12 layers. Despite both models having 12 transformer layers, the same hidden size (768), the same patch size (16), and being pretrained and fine-tuned on ImageNet, the layer-wise distribution of top-attribution neurons is almost inverted between the supervised ViT-B/16 and the self-supervised MAE-B/16. This suggests that the pretraining paradigm fundamentally changes how knowledge is organized across depth.
Evidence
observational
Caveat
The comparison is limited to two models of the same size; the paper does not test whether the inversion holds across other sizes or architectures.