SourceNumber Cookbook: Number Understanding of Language Models and How to Improve It
All tested released LLMs perform well on integer and float tasks but their accuracy drops sharply when the numerical representation changes to fractions or scientific notation. Even in the shortest s range (1-4 digits), average accuracy on these less common representations falls below 20%. This holds for GPT-4o, GPT-4o-mini, Qwen2-72B, and all other tested models. The paper attributes this to insufficient exposure to diverse numerical representations in training data, highlighting a mismatch between models' strong performance on standard integer arithmetic and their poor handling of everyday numerical diversity.