SourceU-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models
The authors evaluate 56 released LLMs across 19 model families on 6 multiple-choice benchmarks (MMLU, Arithmetic, Persian-QA, Hindu Knowledge, Conceptual Combinations, Analogical Similarity), grouping questions by difficulty level using a target-conditioned Brier score. Performance on the hardest question group shows a U-shaped curve: it first worsens with scale (models are distracted by surface patterns such as negations) before improving past the emergence threshold. Performance on the easiest group shows an inverted-U followed by steady improvement, consistent with deep double descent. Because the two trends offset each other when aggregated, overall accuracy stagnates until the easy-question curve reverts to standard scaling, at which point both groups improve and overall performance soars. The pattern is absent on non-emergent tasks (ARC, HellaSwag, Abstract Narrative Understanding), where performance improves monotonically.