SourceRobotouille: An Asynchronous Planning Benchmark for LLM Agents
The paper measures the normalised steps-to-go at the end of failed GPT-4o ReAct runs. In the asynchronous dataset, 58.6% of failures fall in the (0.5, 1.0] bucket, meaning the agent made little to no progress toward the goal before running out of steps. In the synchronous dataset, 41.5% fall in the same bucket, while 45.3% fall in (1.0, ∞), meaning the agent moved away from the goal. The histogram of repeated transitions shows that the lower and upper quartiles for async failures are 103.1% and 55.8% larger than for sync failures, indicating the agent gets stuck in loops more often in the async setting. A recovery annotation shows only 40.4% of async transition failures recover, versus 58.8% in sync.