IC-438All 10 LLM-based agents perform poorly at recognizing when they need to request input from the system or user, with overall accuracy between 30.55% and 55.18%
Haiyang SHEN, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, Yun Ma
The paper evaluates whether agents can identify parameters that require input from the system (clipboard, file, current date) or from the user (ask). Across all 10 LLMs and 4 difficulty levels, recognition accuracy is low. The best overall accuracy is 55.18% (DeepSeek-2-Coder) and the worst is 30.55% (GPT-3.5-turbo). Performance does not improve consistently with model size or intelligence level, and accuracy generally declines as task difficulty increases for most models. This indicates a substantial lack of awareness in existing agents regarding when to request necessary input.
Evidence
correlational
Key metric
overall accuracy ranging between 30.55% (gpt-3.5-turbo) and 55.18% (deepseek-2-coder); at the hardest level (15,30], accuracy drops to 8.11% (qwen-2-57b) and 17.14% (gpt-4o-mini)
Caveat
Evaluation is limited to the specific parameter types defined in Apple Shortcuts (clipboard, extensioninput, currentdate, ask); the metric measures proportion of correctly identified parameters