IC-114Released LLMs cannot reliably identify a specific digit in a number as the number's length increases, with GPT-4o achieving only 20% on get-digit in the xl range
Haotong Yang, Yi Hu, Shijia Kang, Zhouchen Lin, Muhan Zhang
On the 'get digit' task, where the model must return the i-th digit of a given number, all tested released LLMs show significant performance degradation as the number lengthens. In the xl range (15-20 digits), GPT-4o achieves only 20% accuracy, barely above the 10% random-guessing baseline (since the answer is always a single digit 0-9). Performance is good in the shorter s range, indicating models comprehend the task instruction but cannot reliably index into long digit sequences. The paper identifies this as a fundamental limitation that may explain broader numerical processing failures.
Evidence
correlational
Key metric
GPT-4o get-digit xl range: 20% accuracy vs 10% random guessing baseline
Caveat
Performance is good in the shorter s-range, indicating the models can comprehend the task instruction; the failure is specific to longer numbers.