IC-346Hierarchical and categorical concepts from WordNet are linearly represented in the final-layer space of Gemma-2b and Llama-3-8B, with semantic hierarchy encoded as orthogonality and categorical concepts as polytopes
Kiho Park, Yo Joong Choe, Yibo Jiang, Victor Veitch
The paper estimates vector representations for 900+ WordNet synsets (593 noun, 364 verb) in the final-layer unembedding space of Gemma-2b and Llama-3-8B using a variant of LDA. Projections of target words onto the estimated vectors are near 1 while random words are near 0, confirming the existence of vector representations. Cosine similarity between representation vectors mirrors the WordNet hyponym graph structure, and the cosine similarity between a child-parent difference vector and the parent vector is close to 0, as predicted by the hierarchical orthogonality theorem. An interventional check (Table 1) shows that adding the parent contrast vector to context embeddings shifts parent-concept logit differences by 5.1–14.4 while child-concept logit differences remain near 0 (−0.15 to 0.38). Categorical concepts such as {mammal, bird, fish} form polytopes whose vertices are the individual element vectors.
The whitening transformation used to define the causal inner product only applies to the final layer; the geometry of internal layers is not addressed. The authors note that set inclusion in the estimation procedure can produce spurious orthogonality, though they show the effect persists when set inclusion is broken by using independent 70% training subsets per synset.