SourceNumber Cookbook: Number Understanding of Language Models and How to Improve It
On the 'get digit' task, where the model must return the i-th digit of a given number, all tested released LLMs show significant performance degradation as the number lengthens. In the xl range (15-20 digits), GPT-4o achieves only 20% accuracy, barely above the 10% random-guessing baseline (since the answer is always a single digit 0-9). Performance is good in the shorter s range, indicating models comprehend the task instruction but cannot reliably index into long digit sequences. The paper identifies this as a fundamental limitation that may explain broader numerical processing failures.