Within model families, larger variants reach their peak ID-OOD gap (critical complexity) at higher algorithmic complexity levels than smaller variants. For example, Qwen 1.5 shifts from O(n) at 1.8B to O([n³, n⁴]) at 110B; Llama 3.1 shifts from O([n, n²]) at 8B to O([n³, n⁴]) at 405B; Gemma 2 shifts from O([n², n³]) at 2B to O([n³, n⁴]) at 27B. Among closed-source models, Claude 3 Sonnet reaches O([n³, n⁴]) while Haiku reaches O([n², n³]). Notably, GPT-4o-mini (smaller than GPT-4o) exhibits a higher critical complexity of O(2^n), deviating from the size-based trend. O1-mini also reaches O(2^n), the highest observed.
Evidence
correlational
Key metric
Qwen1.5-1.8b critical complexity O(n) vs Qwen1.5-110b O([n³, n⁴]); Llama-3.1-8b O([n, n²]) vs Llama-3.1-405b O([n³, n⁴]); Gemma-2-2b O([n², n³]) vs Gemma-2-27b O([n³, n⁴]); Claude-3-Haiku O([n², n³]) vs Claude-3-Sonnet O([n³, n⁴]); GPT-4o-mini O(2^n) vs GPT-4o O([n³, n⁴]); O1-mini O(2^n).
Caveat
GPT-4o-mini deviates from the expected size-based trend, exhibiting a higher critical complexity than the larger GPT-4o. The authors hypothesize this is due to more aggressive training on high-quality data. Even the largest models eventually reach critical complexity, so scaling delays but does not eliminate over-reliance on memorization.