Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Robotouille: An Asynchronous Planning Benchmark for LLM Agents
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-221
GPT-4o ReAct success rate drops from 47% on synchronous to 11% on asynchronous planning tasks, and all other tested LLMs show equal or worse performance
IC-222
GPT-4o ReAct failures are dominated by rule violations (transition function) and goal misinterpretation, with the balance shifting from goal-dominant in synchronous to transition-dominant in asynchronous settings
IC-223
GPT-4o ReAct shows poor recovery from failures in asynchronous settings, with 58.6% of failed runs making little to no progress toward the goal and significantly higher repeated transitions than in synchronous settings
IC-224
GPT-4o ReAct cannot incorporate stochastic state changes, with success rate on cutting tasks dropping from 56% to 1% when a 33% chance of a cut item reverting to uncut is introduced