Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-298
Weight similarity in open-source LLMs is organized in a depth-dependent structure with adjacent-layer similarity and distinct clusters at specific depths
IC-299
Instruction tuning preserves the weight-matrix structure of LLMs, with DOCS scores exceeding 0.7 across all matrices
IC-300
One MoE expert in Mixtral-8x7B is structurally distinct from the others in many layers
IC-301
Yi-1.5-9B-chat exhibits a layer-repetition pattern where a section of layers is duplicated at a later depth