IC-434The critical complexity at which LLMs over-rely on memorization shifts to higher task difficulty as model size increases

Zhenting Qi, Hongyin Luo, Xuliang Huang, Zhuokai Zhao, Yibo Jiang, Xiangjun Fan, Himabindu Lakkaraju, James R. Glass

SourceQuantifying Generalization Complexity for Large Language Models

Within model families, larger variants reach their peak ID-OOD gap (critical complexity) at higher algorithmic complexity levels than smaller variants. For example, Qwen 1.5 shifts from O(n) at 1.8B to O([n³, n⁴]) at 110B; Llama 3.1 shifts from O([n, n²]) at 8B to O([n³, n⁴]) at 405B; Gemma 2 shifts from O([n², n³]) at 2B to O([n³, n⁴]) at 27B. Among closed-source models, Claude 3 Sonnet reaches O([n³, n⁴]) while Haiku reaches O([n², n³]). Notably, GPT-4o-mini (smaller than GPT-4o) exhibits a higher critical complexity of O(2^n), deviating from the size-based trend. O1-mini also reaches O(2^n), the highest observed.

Evidence
correlational
Key metric
Qwen1.5-1.8b critical complexity O(n) vs Qwen1.5-110b O([n³, n⁴]); Llama-3.1-8b O([n, n²]) vs Llama-3.1-405b O([n³, n⁴]); Gemma-2-2b O([n², n³]) vs Gemma-2-27b O([n³, n⁴]); Claude-3-Haiku O([n², n³]) vs Claude-3-Sonnet O([n³, n⁴]); GPT-4o-mini O(2^n) vs GPT-4o O([n³, n⁴]); O1-mini O(2^n).
Caveat
GPT-4o-mini deviates from the expected size-based trend, exhibiting a higher critical complexity than the larger GPT-4o. The authors hypothesize this is due to more aggressive training on high-quality data. Even the largest models eventually reach critical complexity, so scaling delays but does not eliminate over-reliance on memorization.
Model
Qwen1.5 1.8B, Qwen1.5-4B, Qwen1.5-7B, 14B, Qwen1.5-32B, Qwen1.5-72B, Qwen1.5-110B, Llama 3.1 8B, 70B, 405B, Gemma 2 2B, 9B, 27B, Claude 3 Haiku, Sonnet, GPT-4o mini, O1 / OpenAI-o1-preview O1-mini
Concepts
Scale-dependent behaviour
Methods
Zero-shot Chain-of-Thought / Wei et al. 2022 (Chain of Thought) / Zero-shot chain-of-thought prompting [primary]
Related findings
IC-433, IC-435
Extraction
automatic-extraction