SourceDOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models
Applying DOCS to the MLP-up weight matrices of 01-ai/yi-1.5-9b-chat reveals a specific off-diagonal pattern: layer 9 is highly similar to layer 25, layer 10 to layer 26, layer 11 to layer 27, and so on. This suggests a repetition of a section of layers within the model, possibly due to a specific training strategy employed to save training costs. In contrast, representational similarity (Linear CKA) on the same model shows relatively homogeneous patterns that do not reveal this fine-grained layer correspondence.