IC-451Transformer block coupling of Jacobian singular vectors positively correlates with benchmark performance across 30+ LLMs, more strongly than parameter count, depth, or embedding dimension
Murdock Aubry, Haoming Meng, Anton Sugolov, Vardan Papyan
The paper computes Jacobians of each transformer block along token trajectories, performs SVD, and measures the alignment of top-k singular vectors between blocks (coupling). Across 30+ released LLMs from over 8 organizations, the average coupling score correlates with HuggingFace Open LLM Leaderboard benchmark scores. The regression fit yields R-squared of 0.8 with p-value 9.99e-10. This correlation exceeds the correlation of benchmark score with parameter count, model depth, or embedding dimension. The relationship is robust across different choices of k (1%, 3%, 5% of hidden dimension).
Evidence
correlational
Key metric
R^2 = 0.8, p = 9.99 x 10^-10 (coupling vs. benchmark score, k = 1/10 d_model); correlation stronger than with parameter count, model depth, or embedding dimension
Caveat
Coupling is measured on a smaller subset of dataset prompts due to computational cost of Jacobian computation; the correlation is across models, not causal.