SourceArithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics
The authors test two hypotheses for why the bag of heuristics produces incorrect answers: (i) the prompt lacks sufficient associated heuristic neurons, or (ii) the heuristics have imperfect recall or low logits for the correct answer token. Sampling 50 correct and 50 incorrect prompts, they find that incorrect prompts actually have MORE associated heuristic neurons on average, ruling out hypothesis (i). However, the total logit contribution of all heuristic neurons to the correct answer token is slightly higher for correct prompts than for incorrect ones, supporting hypothesis (ii): the primary failure mode is poor promotion of the correct answer logit, not an absence of heuristics.