SourceBeyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution
The paper measures average cosine similarity between concept vectors for 16 low-level concepts grouped into 4 high-level categories (sports events, populated places, animals, movie genres) at the penultimate layer of three LLMs. In all three models, concepts within the same high-level category show substantially higher cosine similarity than concepts from different categories, forming distinct 4x4 blocks in the similarity matrix. PCA projections of the concept vectors confirm this clustering. Additionally, Llama-2-7B and Llama-2-13B show stronger cross-category correlations between populated places and animals (e.g., island correlates more with bird and fish than with insect and cat), aligning with real-world semantic relationships.