IC-221GPT-4o ReAct success rate drops from 47% on synchronous to 11% on asynchronous planning tasks, and all other tested LLMs show equal or worse performance
Gonzalo Gonzalez-Pumariega, Leong Su Yean, Neha Sunkara, Sanjiban Choudhury
The paper evaluates multiple LLMs using three agent baselines (I/O, I/O CoT, ReAct) on the Robotouille benchmark's synchronous and asynchronous cooking-planning datasets. GPT-4o with ReAct is the best performer, achieving 47% on synchronous tasks but only 11% on asynchronous tasks. GPT-4o-mini drops from 11% to 0%, Gemini-1.5-Flash scores 0% on both, and Claude-3-Haiku scores 2% on both. Open-source models (Qwen2-72B, Llama-3.1-70B, Gemma-2-27B) score at most 7% on synchronous and 2% on asynchronous. The gap is confirmed by a controlled comparison where the first three async tasks are made synchronous (delay set to 0), and GPT-4o ReAct improves from 20-40% to 50-60%.
Success is defined as reaching the goal within 1.5 times the optimal number of steps; the benchmark has only 10 tasks per dataset with 10 procedurally generated instances each, and a single in-context example is provided.