IC-137Pre-trained ResNet34 and ViT-B features on CIFAR-100 exhibit a block-diagonal class-correlation structure, with ViT-B showing higher intra-class correlation (0.35) than ResNet34 (0.25)
The paper measures pairwise feature correlations of pre-trained ResNet34 and ViT-B on CIFAR-100 to verify that their features satisfy the block-diagonal gram matrix structure assumed in their theory. For ResNet34, the mean intra-class feature correlation is 0.25±0.16, super-class correlation is 0.02±0.12, and inter-superclass correlation is -0.03±0.10. For ViT-B, the intra-class correlation rises to 0.35, which the authors note amplifies the clustering effect of self-distillation so that most gains appear in the first two distillation rounds. The same block-diagonal pattern is confirmed on five additional datasets (Caltech-101, Caltech-256, Flowers-102, Food-101, StanfordCars) for ResNet34.
Evidence
observational
Key metric
intra-class feature correlation 0.25±0.16 (ResNet34) vs 0.35 (ViT-B) on CIFAR-100; super-class 0.02±0.12; inter-superclass -0.03±0.10 (ResNet34, Table 2)
Caveat
The ViT-B intra-class correlation (0.35) is reported only for CIFAR-100; full per-dataset statistics (Table 2) are provided only for ResNet34. The measurements are on the full training set, not a held-out split.