anchor
Findings
- IC-072LLM agents of varying scales exhibit a failure mode on web automation tasks when processing raw, complex web page observations, with the penalty being more severe for smaller models [compared-to]
- IC-1003Pretrained ResNet-50 and ViT-B/16 exhibit neuron activation patterns that are separable between in-distribution and out-of-distribution inputs, enabling post-hoc OOD detection without model modification [compared-to]
- IC-1193Vicuna and Alpaca achieve 0% pass rate on all ToolBench tool-use instructions, while GPT-4 and ChatGPT reach 71.1% and 64.8% with DFSDT, revealing a wide capability gap in tool use among released LLMs [compared-to]
- IC-1330All 20 evaluated LLMs improve in multi-turn task-solving with additional tool-use turns and GPT-4-simulated language feedback [builds-on]
- IC-1331SIFT and RLHF variants of CodeLlama and Llama-2 perform worse than their base counterparts in multi-turn interaction [builds-on]
- IC-221GPT-4o ReAct success rate drops from 47% on synchronous to 11% on asynchronous planning tasks, and all other tested LLMs show equal or worse performance [builds-on]
- IC-221GPT-4o ReAct success rate drops from 47% on synchronous to 11% on asynchronous planning tasks, and all other tested LLMs show equal or worse performance [primary]
- IC-222GPT-4o ReAct failures are dominated by rule violations (transition function) and goal misinterpretation, with the balance shifting from goal-dominant in synchronous to transition-dominant in asynchronous settings [primary]
- IC-223GPT-4o ReAct shows poor recovery from failures in asynchronous settings, with 58.6% of failed runs making little to no progress toward the goal and significantly higher repeated transitions than in synchronous settings [primary]
- IC-224GPT-4o ReAct cannot incorporate stochastic state changes, with success rate on cutting tasks dropping from 56% to 1% when a 33% chance of a cut item reverting to uncut is introduced [primary]
- IC-436API selection accuracy of 10 LLM-based agents degrades sharply as task complexity increases, with open-source models ≥70B matching closed-source on simpler tasks but lagging on the most complex [primary]
- IC-437Extracting parameters from user queries is harder for LLM-based agents than using outputs from previous actions, and less intelligent LLMs show steeper parameter-filling degradation with task difficulty [primary]
- IC-438All 10 LLM-based agents perform poorly at recognizing when they need to request input from the system or user, with overall accuracy between 30.55% and 55.18% [primary]
- IC-439Agent-specialized fine-tuned models (XLAM) significantly improve API selection over base models, but code-fine-tuned models (AgentLM) degrade performance, and no fine-tuning approach improves input recognition [primary]