IC-148Language models represent semantically equivalent inputs from different data types (languages, code, images, audio) close together in intermediate layers, with the shared space scaffolded by the model's dominant language
The paper tests whether LMs map semantically similar but surface-distinct inputs into a shared representation space. Using cosine similarity on parallel English-Chinese texts, logit lens on Chinese/English/code/arithmetic inputs, and patch-level logit lens on images and audio, the authors show that in intermediate layers (roughly layers 10-25 of 32), representations of equivalent inputs from different data types are significantly closer than to non-equivalent inputs. The logit lens further reveals that the shared space is anchored by the dominant language: when Llama-3 processes Chinese, English tokens have higher logit lens probability than Chinese tokens from layer 17 onward; when Baichuan-2 processes English, Chinese tokens dominate in intermediate layers. For code, the 'and' token outperforms the actual next Python token in middle layers; for images, the corresponding color word is the closest token >20% of the time; for audio, matching label words outperform random labels.
Evidence
correlational
Key metric
cosine similarity up to >80% for parallel English-Chinese texts in middle layers; logit lens: English token probability exceeds Chinese token from layer 17 to 31 (Llama-3); 'and' token logit lens probability exceeds next-token probability in middle layers for Python list literals (n=6923); closest token is the corresponding color word in >20% of intermediate-layer patches (n=2304); matching caption nouns have higher logit lens alignment than irrelevant captions across all layers for LLaVA
Caveat
For BLOOM, a multilingual model with balanced training, no clear dominant language is observed in intermediate layers; the closest tokens are often symbols with no clear semantics. The cosine similarity for image-caption pairs is lower than for mutual translations. The logit lens is a training-free probe and may not capture the full structure of the representation.