IC-451Transformer block coupling of Jacobian singular vectors positively correlates with benchmark performance across 30+ LLMs, more strongly than parameter count, depth, or embedding dimension
The paper computes Jacobians of each transformer block along token trajectories, performs SVD, and measures the alignment of top-k singular vectors between blocks (coupling). Across 30+ released LLMs from over 8 organizations, the average coupling score correlates with HuggingFace Open LLM Leaderboard benchmark scores. The regression fit yields R-squared of 0.8 with p-value 9.99e-10. This correlation exceeds the correlation of benchmark score with parameter count, model depth, or embedding dimension. The relationship is robust across different choices of k (1%, 3%, 5% of hidden dimension).
Evidence
correlational
Key metric
R^2 = 0.8, p = 9.99 x 10^-10 (coupling vs. benchmark score, k = 1/10 d_model); correlation stronger than with parameter count, model depth, or embedding dimension
Caveat
Coupling is measured on a smaller subset of dataset prompts due to computational cost of Jacobian computation; the correlation is across models, not causal.