IC-437Extracting parameters from user queries is harder for LLM-based agents than using outputs from previous actions, and less intelligent LLMs show steeper parameter-filling degradation with task difficulty
Haiyang SHEN, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, Yun Ma
The paper measures API parameter value filling across 4 difficulty levels for 10 LLMs, separating primitive/enum data type filling from filling outputs of previous actions. Across all models, the accuracy for primitive/enum filling is lower than for previous-action output filling, with the gap ranging from 2.55% (GPT-3.5-turbo) to 15.39% (DeepSeek-2-Chat). For less intelligent LLMs like GPT-4o-mini, parameter filling accuracy drops significantly as difficulty increases, while more intelligent LLMs like Gemini-1.5-Pro show smaller drops. Error analysis shows that less intelligent models frequently make format errors and wrong API selections, while powerful models rarely make format errors.
Evidence
correlational
Key metric
accuracy drop between primitive/enum fill and previous-actions fill ranges from 2.55% (gpt-3.5-turbo) to 15.39% (deepseek-2-chat); general LLMs achieve parameter filling accuracies of 94.29%, 83.92%, and 75.53% across three difficulty levels
Caveat
Evaluation uses a modified ReAct prompt template; parameter values are checked against human-annotated golden action sequences