IC-112Released LLMs show a reproducible failure mode where numerical task accuracy degrades sharply as input digit length increases

Haotong Yang, Yi Hu, Shijia Kang, Zhouchen Lin, Muhan Zhang

SourceNumber Cookbook: Number Understanding of Language Models and How to Improve It

Across all eight tested released LLMs, accuracy on numerical tasks (addition, comparison, digit operations) drops substantially as the number of digits in the input grows. GPT-4o's integer addition accuracy falls from nearly 100% in the s range (1-4 digits) to around 40% in the l range (9-14 digits) and just 15% in the xl range (15-20 digits). The same pattern holds for float addition (90% to <5%) and is consistent across Llama-3.1, Qwen2, Mixtral, and Llama-2. The degradation affects both per-digit accuracy (digit match) and overall answer length (dlength), indicating the model struggles with both individual digit processing and generating correctly-lengthed outputs.

Evidence

correlational

Key metric

GPT-4o integer addition: nearly 100% (s), 80% (m), around 40% (l), just 15% (xl); GPT-4o float addition: 90% (s), 60% (m), 15% (l), less than 5% (xl); Qwen2 integer-length: almost 100% (s), 50% (m), below 5% (l and xl)

Caveat

Results are zero-shot; 5-shot prompting provides some improvement but the length degradation pattern persists. The paper notes the number of models tested is limited.

Model

GPT-4o mini, Llama 3.1 8B, 70B, Llama 2 / Llama 2 base Llama-2-7B-HF, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Qwen 2 7B, 72B

Concepts

Failure mode

Related findings

IC-113, IC-114, IC-115

Extraction

automatic-extraction