IC-43256 LLMs from 19 families exhibit u-shaped scaling on hard questions and inverted-U scaling on easy questions, with the opposing trends explaining emergent ability stagnation
The authors evaluate 56 released LLMs across 19 model families on 6 multiple-choice benchmarks (MMLU, Arithmetic, Persian-QA, Hindu Knowledge, Conceptual Combinations, Analogical Similarity), grouping questions by difficulty level using a target-conditioned Brier score. Performance on the hardest question group shows a U-shaped curve: it first worsens with scale (models are distracted by surface patterns such as negations) before improving past the emergence threshold. Performance on the easiest group shows an inverted-U followed by steady improvement, consistent with deep double descent. Because the two trends offset each other when aggregated, overall accuracy stagnates until the easy-question curve reverts to standard scaling, at which point both groups improve and overall performance soars. The pattern is absent on non-emergent tasks (ARC, HellaSwag, Abstract Narrative Understanding), where performance improves monotonically.
The authors note that emergent phenomena are common across LLM benchmarks and it might be hard to claim all of them show clear u-shaped vs. inverted-u scaling. The focus is on multiple-choice tasks; applying the method to string-matching tasks requires identifying a suitable continuous metric.