IC-427Newer base models (post-November 2023) outperform older ones by 7.3 points on MMLU and 19.1 points on GSM8K controlling for pretraining compute, but this gap vanishes after fine-tuning all models on the same task-relevant data
Ricardo Dominguez-Olmedo, Florian E. Dorner, Moritz Hardt
The paper evaluates 56 base models (70M–70B parameters) on MMLU and GSM8K, splitting them into those trained before and after November 2023. Controlling for pretraining compute via a log-linear regression, newer models show a statistically significant advantage of 0.073 on MMLU and 0.191 on GSM8K. After fine-tuning all models on the same task-specific data (MMLU auxiliary set for MCQA, MetaMathQA+Orca-Math for math), the estimated difference drops to 0.005 for both benchmarks and is no longer statistically significant. The same pattern appears when ARC and HellaSwag are reformulated as multiple-choice questions (differences of 0.120 and 0.114 unadjusted, 0.014 and 0.009 adjusted), indicating the confound is about task format rather than data contamination.
Evidence
interventional
Key metric
MMLU: difference = 0.073 (unadjusted, R²=0.943) vs 0.005 (adjusted, R²=0.989); GSM8K: difference = 0.191 (unadjusted, R²=0.901) vs 0.005 (adjusted, R²=0.958); ARC MCQA: 0.120 vs 0.014; HellaSwag MCQA: 0.114 vs 0.009
Caveat
The fine-tuning uses a fixed set of hyperparameters; the authors verify robustness by giving newer models a learning-rate search advantage, which does not change the conclusion. The causal interpretation assumes test task training saturates with sufficient fine-tuning data.