IC-1544The latent spaces of pretrained foundational models across vision and text are not related by a single class of geometric transformations; the optimal alignment depends on the specific model pair, architecture, and dataset.
Irene Cannistraci, Luca Moschella, Marco Fumero, Valentino Maiorca, Emanuele Rodolà
The paper measures CKA similarity between the latent spaces of eleven pretrained foundational models (five vision, six text) under four different invariance assumptions (cosine/angle-preserving, Euclidean/translation-invariant, L1, L∞). No single projection consistently yields the highest similarity across all model pairs and datasets. For example, on CIFAR-10, the best projection differs between ViT-based models and CLIP. In zero-shot stitching, the L∞ projection is consistently worst (e.g., 0.31 accuracy for ViT-B/16 on CIFAR-100) while cosine, Euclidean, and L1 are comparable (0.77–0.79), and the product of all four achieves the highest score (0.81). This demonstrates that the geometric structure relating these models' representations is not captured by any single transformation class.
Evidence
correlational
Key metric
ViT-B/16 on CIFAR-100 zero-shot stitching accuracy: cosine 0.79±0.03, euclidean 0.79±0.03, l1 0.77±0.04, l∞ 0.31±0.03, product of all four 0.81±0.04; CLIP ViT-B/32 on CIFAR-100: cosine 0.52±0.03, euclidean 0.53±0.02, l1 0.53±0.04, l∞ 0.27±0.04, product 0.58±0.03
Caveat
The analysis uses CKA as the similarity metric, which the authors note (citing Davari et al. 2022) is sensitive to shifts in the latent space. The finding is limited to the specific set of models and datasets tested.