IC-113Released LLMs show a reproducible failure mode where accuracy on fraction and scientific notation tasks falls below 20% even for the shortest inputs
Haotong Yang, Yi Hu, Shijia Kang, Zhouchen Lin, Muhan Zhang
All tested released LLMs perform well on integer and float tasks but their accuracy drops sharply when the numerical representation changes to fractions or scientific notation. Even in the shortest s range (1-4 digits), average accuracy on these less common representations falls below 20%. This holds for GPT-4o, GPT-4o-mini, Qwen2-72B, and all other tested models. The paper attributes this to insufficient exposure to diverse numerical representations in training data, highlighting a mismatch between models' strong performance on standard integer arithmetic and their poor handling of everyday numerical diversity.
Evidence
correlational
Key metric
average accuracy falling below 20%, even for the shortest s-range (1-4 digits) on fraction and scientific notation tasks
Caveat
The paper acknowledges the range of tasks could still be incomplete and biased toward certain aspects. The number of models tested is limited.