IC-427Newer base models (post-November 2023) outperform older ones by 7.3 points on MMLU and 19.1 points on GSM8K controlling for pretraining compute, but this gap vanishes after fine-tuning all models on the same task-relevant data
The paper evaluates 56 base models (70M–70B parameters) on MMLU and GSM8K, splitting them into those trained before and after November 2023. Controlling for pretraining compute via a log-linear regression, newer models show a statistically significant advantage of 0.073 on MMLU and 0.191 on GSM8K. After fine-tuning all models on the same task-specific data (MMLU auxiliary set for MCQA, MetaMathQA+Orca-Math for math), the estimated difference drops to 0.005 for both benchmarks and is no longer statistically significant. The same pattern appears when ARC and HellaSwag are reformulated as multiple-choice questions (differences of 0.120 and 0.114 unadjusted, 0.014 and 0.009 adjusted), indicating the confound is about task format rather than data contamination.
Evidence
interventional
Key metric
MMLU: difference = 0.073 (unadjusted, R²=0.943) vs 0.005 (adjusted, R²=0.989); GSM8K: difference = 0.191 (unadjusted, R²=0.901) vs 0.005 (adjusted, R²=0.958); ARC MCQA: 0.120 vs 0.014; HellaSwag MCQA: 0.114 vs 0.009
Caveat
The fine-tuning uses a fixed set of hyperparameters; the authors verify robustness by giving newer models a learning-rate search advantage, which does not change the conclusion. The causal interpretation assumes test task training saturates with sufficient fine-tuning data.