IC-1171Pre-trained ViT, MAE, and ResNet50 (supervised and MoCo v2) place visually similar but semantically distinct ImageNet classes (mop, broom, puck, crutch) in close proximity in their feature space

Ilmin Kang, HyounYoung Bae, Kangil Kim

SourceLabel-Focused Inductive Bias over Latent Object Features in Visual Classification

The paper extracts class-level feature centroids from pre-trained ViT, MAE, and ResNet50 (both supervised and contrastive/MoCo v2) on ImageNet-1K. In all four models, classes that share visual appearance but differ semantically—mop and broom (shared stick), puck and crutch (shared cylindrical shape)—have centroids that are closely located in the final-layer feature space. This indicates the representations are dominated by input-domain visual similarity rather than by the semantic structure of the labels. The effect persists under contrastive pre-training (MoCo v2), which only slightly decouples some affected pairs but still fails to separate broom from puck.

Evidence

observational

Caveat

The observation is qualitative (t-SNE visualizations of a small number of example classes) and not quantified across all 1000 ImageNet classes; the paper does not report a distance metric or the fraction of class pairs affected.

Model

ViT, MAE, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN ResNet50

Datasets

ImageNet-1k / ImageNet / ImageNet-1k-val / ImageNet-Val [eval]

Methods

t-SNE [primary]

Related work

MoCo v2 [compared-to]

Extraction

automatic-extraction