CKA

anchor

Findings

• IC-1540CLS-token attention maps in pretrained ViT-t/16 exhibit high inter-layer correlation (cosine similarity up to 0.97) concentrated in layers 3–10, and MSA block outputs show high CKA in layers 2–8 [primary]
• IC-1544The latent spaces of pretrained foundational models across vision and text are not related by a single class of geometric transformations; the optimal alignment depends on the specific model pair, architecture, and dataset. [eval]
• IC-548CLIP-B/32 exhibits progressively increasing layer-wise representation similarity in both its vision encoder and text encoder, and the pattern also holds across modalities [compared-to]