SourceDOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models
The paper compares weight matrices of base and instruction-tuned models in the Yi-1.5, Llama-3.1, and Gemma-2 families using the DOCS index. All DOCS scores between corresponding layers of the base and instruct variants exceed 0.7, indicating that instruction fine-tuning largely retains the foundational weight structure. Additionally, the weight matrices cluster into three functional groups based on their DOCS score trends: MLP-up with MLP-down, Wq with Wk, and Wv with Wo.