SourceThe Geometry of Categorical and Hierarchical Concepts in Large Language Models
The paper estimates vector representations for 900+ WordNet synsets (593 noun, 364 verb) in the final-layer unembedding space of Gemma-2b and Llama-3-8B using a variant of LDA. Projections of target words onto the estimated vectors are near 1 while random words are near 0, confirming the existence of vector representations. Cosine similarity between representation vectors mirrors the WordNet hyponym graph structure, and the cosine similarity between a child-parent difference vector and the parent vector is close to 0, as predicted by the hierarchical orthogonality theorem. An interventional check (Table 1) shows that adding the parent contrast vector to context embeddings shifts parent-concept logit differences by 5.1–14.4 while child-concept logit differences remain near 0 (−0.15 to 0.38). Categorical concepts such as {mammal, bird, fish} form polytopes whose vertices are the individual element vectors.