TM-002The mean-difference concept vector scores higher than trained classifiers under cosine similarity

Filip Fedor, Jan Urbanek, Krzysztof Czerniawski

SourceConcept vectors in TerraMind latent space

The subtraction construction from TM-001 scores higher than a logistic regression and a linear SVM trained to capture the same concept under the report's cosine-similarity evaluation. The authors caution that the classifiers' lower values likely stem from this evaluation setup and may not persist under another metric.

Evidence
correlational
Key metric
cosine similarity to the empirical shift, mostly burned and winter: mean difference 0.713 and 0.799, logistic regression 0.503 and 0.725, linear SVM 0.361 and 0.604
Caveat
The authors state that the lower classifier scores likely stem from the evaluation metric and identify the lack of alternative evaluation methods as the main limitation of the comparison.
Model
TerraMind v1 base
Concepts
Method artefact, Linear representation
Datasets
Copernicus Data Space Ecosystem [source]
Methods
Mean-difference concept vector [primary], Logistic regression / Linear decoder (logistic regression) / Logistic classification / Logistic regression head [compared-to], Linear support vector machine [compared-to], Cosine similarity / Cosine similarity analysis / Cosine semantic similarity / cosine similarity of hidden states / Sample-wise cosine similarity / Cosine similarity of attention maps / Cosine similarity perturbation analysis / Cosine similarity template matching / Cosine similarity to neighbours / Semantic consistency (cosine similarity) [eval]
Related work
TCAV (Testing with Concept Activation Vectors) [builds-on]
Related findings
TM-001
Extraction
automatic-extraction