IC-438All 10 LLM-based agents perform poorly at recognizing when they need to request input from the system or user, with overall accuracy between 30.55% and 55.18%
The paper evaluates whether agents can identify parameters that require input from the system (clipboard, file, current date) or from the user (ask). Across all 10 LLMs and 4 difficulty levels, recognition accuracy is low. The best overall accuracy is 55.18% (DeepSeek-2-Coder) and the worst is 30.55% (GPT-3.5-turbo). Performance does not improve consistently with model size or intelligence level, and accuracy generally declines as task difficulty increases for most models. This indicates a substantial lack of awareness in existing agents regarding when to request necessary input.
Evidence
correlational
Key metric
overall accuracy ranging between 30.55% (gpt-3.5-turbo) and 55.18% (deepseek-2-coder); at the hardest level (15,30], accuracy drops to 8.11% (qwen-2-57b) and 17.14% (gpt-4o-mini)
Caveat
Evaluation is limited to the specific parameter types defined in Apple Shortcuts (clipboard, extensioninput, currentdate, ask); the metric measures proportion of correctly identified parameters