IC-137Pre-trained ResNet34 and ViT-B features on CIFAR-100 exhibit a block-diagonal class-correlation structure, with ViT-B showing higher intra-class correlation (0.35) than ResNet34 (0.25)

Hyeonsu Jeong, Hye Won Chung

SourceRethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels

The paper measures pairwise feature correlations of pre-trained ResNet34 and ViT-B on CIFAR-100 to verify that their features satisfy the block-diagonal gram matrix structure assumed in their theory. For ResNet34, the mean intra-class feature correlation is 0.25±0.16, super-class correlation is 0.02±0.12, and inter-superclass correlation is -0.03±0.10. For ViT-B, the intra-class correlation rises to 0.35, which the authors note amplifies the clustering effect of self-distillation so that most gains appear in the first two distillation rounds. The same block-diagonal pattern is confirmed on five additional datasets (Caltech-101, Caltech-256, Flowers-102, Food-101, StanfordCars) for ResNet34.

Evidence

observational

Key metric

intra-class feature correlation 0.25±0.16 (ResNet34) vs 0.35 (ViT-B) on CIFAR-100; super-class 0.02±0.12; inter-superclass -0.03±0.10 (ResNet34, Table 2)

Caveat

The ViT-B intra-class correlation (0.35) is reported only for CIFAR-100; full per-dataset statistics (Table 2) are provided only for ResNet34. The measurements are on the full training set, not a held-out split.

Model

ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN ResNet-34, ViT ViT-B

Concepts

Distance preservation

Datasets

CIFAR-100 [eval], Caltech-101 [eval], Caltech-256 [eval], Flowers-102 [eval], Food-101 [eval], Stanford Cars [eval]

Extraction

automatic-extraction