IC-298Weight similarity in open-source LLMs is organized in a depth-dependent structure with adjacent-layer similarity and distinct clusters at specific depths
Using the DOCS index, the paper measures cosine-similarity distributions between weight matrices (Wv, Wk, Wq, Wo, MLP-up, MLP-down) across all transformer layers of several released LLMs. Adjacent layers consistently show higher similarity than distant ones, and similarity decays with increasing layer distance. Beyond smooth decay, discrete clusters of mutually similar layers emerge at specific depths: a prominent mid-depth cluster centered around layer 19 in Gemma-2-9B and Gemma-2-27B, and around layer 10 in Llama-3.1-8B and Mixtral-8x7B, plus a second, less pronounced cluster in the final layers. Llama-3.1-70B contains multiple such clusters.
Evidence
observational
Key metric
Gini coefficient for DOCS on MLP-up of Llama 3.1-8B-Instruct: 0.0745 (vs Linear CKA 0.0488, SVCCA 0.0225, CCA 0.0098); cluster in layers 7-12 of Llama 3.1-8B shows 'approximately twice the docs index values of other elements'; mid-depth cluster centered around layer 19 (Gemma-2-9B/27B) and layer 10 (Llama-3.1-8B, Mixtral-8x7B); second cluster after layer 27 (Gemma-2-9B), 33 (Gemma-2-27B), 21 (Llama-3.1-8B), 23 (Mixtral-8x7B)
Caveat
The first and last layers also show higher similarity than expected, hypothesized to be due to their proximity to token embeddings; the authors note this as an exception to the monotonic decay pattern.