ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents

2025-01-22 · ICLR 2025 Poster · anchor

Findings

IC-436API selection accuracy of 10 LLM-based agents degrades sharply as task complexity increases, with open-source models ≥70B matching closed-source on simpler tasks but lagging on the most complex
IC-437Extracting parameters from user queries is harder for LLM-based agents than using outputs from previous actions, and less intelligent LLMs show steeper parameter-filling degradation with task difficulty
IC-438All 10 LLM-based agents perform poorly at recognizing when they need to request input from the system or user, with overall accuracy between 30.55% and 55.18%
IC-439Agent-specialized fine-tuned models (XLAM) significantly improve API selection over base models, but code-fine-tuned models (AgentLM) degrade performance, and no fine-tuning approach improves input recognition