SourceProcedural Knowledge in Pretraining Drives Reasoning in Large Language Models
Comparing the source datasets of the top and bottom k influential documents to the pretraining distribution, code sources are heavily overrepresented for reasoning queries. StackExchange has 10x more influential data than expected for the top portions, and other code sources are 2x overrepresented for k = 50 to 50000. For the 7B, StackExchange and math/trivia have multipliers of 50 and 24 respectively at k=50 for reasoning, versus 27 and 5 for factual. For the 35B, the multipliers are 62 and 21 for reasoning. Content analysis of the top 500 most frequent documents among the k=5000 most influential confirms the 'code' category dominates, while random samples show a much smaller fraction of code.