SourceArithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics
Analyzing Pythia-6.9B training checkpoints from 23k to 143k steps, the authors show that the final heuristic neurons first appear early and gradually converge. The heuristic neurons mutual with the final checkpoint account for an average of 79% of the total heuristics' contribution to accuracy at each intermediate checkpoint, indicating that non-mutual heuristics are less important and become vestigial. Prompt-guided neuron knockout at each checkpoint shows that ablating 25 associated heuristic neurons per layer causes near-zero accuracy at all training stages, confirming the causal link between heuristics and correct completion exists throughout training, not just at the end.