SourceRobotouille: An Asynchronous Planning Benchmark for LLM Agents
The paper categorises all GPT-4o ReAct failures into four MDP-based categories: state, action, transition function, and goal. In the synchronous dataset, goal uncertainty accounts for 64.1% of failures and transition function for 32.1%. In the asynchronous dataset, the balance inverts: transition function accounts for 56.8% and goal for 34.1%. The single most common specific failure in both settings is violating the 'one item at a station' rule, at 24.5% of sync failures and 53.4% of async failures. Adding more rule details to the prompt (ReAct + prior) reduces this specific violation from 38.1% to 22.2% but does not significantly improve overall success rate (0.30 ± 0.085 vs 0.40 ± 0.050).