Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-218
LLaMA3-8B and other LLMs solve arithmetic via a bag of independent heuristic neurons in middle and late MLP layers rather than a robust algorithm
IC-219
The bag-of-heuristics mechanism in LLaMA3-8B fails on certain arithmetic prompts due to insufficient total logit contribution from heuristic neurons, not due to a lack of associated heuristics
IC-220
In Pythia-6.9B, the bag-of-heuristics mechanism emerges gradually during training and is the primary arithmetic mechanism from the earliest checkpoint showing good performance (23k steps)