SourceRobotouille: An Asynchronous Planning Benchmark for LLM Agents
The paper introduces stochasticity into the Robotouille synchronous environment: with a 33% chance, a cut item reverts to uncut once per environment seed. On six synchronous tasks involving cutting, GPT-4o ReAct's success rate drops from 56% in the deterministic setting to 1% in the stochastic setting. The authors note that the rare successes occur when the stochastic change happens immediately, allowing the agent to continue cutting, suggesting the model cannot re-plan after an unexpected state change mid-trajectory.