IC-436API selection accuracy of 10 LLM-based agents degrades sharply as task complexity increases, with open-source models ≥70B matching closed-source on simpler tasks but lagging on the most complex
Haiyang SHEN, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, Yun Ma
The paper evaluates 10 LLMs (5 open-source, 5 closed-source) on API selection across 4 difficulty levels of ShortcutsBench. At the easiest level (0,1] all models perform well, but accuracy drops by an average of 19% moving to (1,5] and by 46% moving to (5,15]. Open-source models ≥70B (Llama-3-70B, Qwen-2-72B, DeepSeek-2-Chat/Coder) match or exceed closed-source models on the first three difficulty levels but fall behind on the hardest (15,30] level. Performance also varies by task type, with lifestyle & social tasks scoring highest and development & API tasks lowest.
Evidence
correlational
Key metric
average accuracy dropped by 19% as task difficulty rose from (0,1] to (1,5], ranging from a 9% decrease (deepseek-2-chat) to a 44% (chatglm-4-air); from (0,1] to (5,15], accuracy fell by 46%, with drops from 38% (gemini-1.5-pro) to 58% (chatglm-4-air); performance difference across task types ranges from 15.94% (gpt-4o-mini) to 36.70% (gemini-1.5-pro)
Caveat
Gemini-1.5-Pro was tested on only 801 instances due to cost, while other models were tested on 5000+ instances; shortcuts longer than 30 actions were excluded from evaluation