IC-072LLM agents of varying scales exhibit a failure mode on web automation tasks when processing raw, complex web page observations, with the penalty being more severe for smaller models
Dongjun Lee, Juyong Lee, Kyuyoung Kim, Jihoon Tack, Jinwoo Shin, Yee Whye Teh, Kimin Lee
Five released LLM agents (GPT-4o, Gemini-1.5-Flash, Claude-3.5-Sonnet, Llama-3.1-8B, Llama-3.1-70B) were evaluated on web automation benchmarks (WebShop, WorkArena, WebArena) using raw accessibility-tree observations. All models show substantially lower success rates on raw observations compared to contextualized ones. The penalty is scale-dependent: Llama-3.1-8B achieves only 1.2% on WorkArena with raw observations, while GPT-4o achieves 38.2% and Claude-3.5-Sonnet 44.8%. On WebShop, raw-observation success rates range from 26.6% (Claude-3.5-Sonnet) to 43.6% (Gemini-1.5-Flash). When observations are contextualized, all models improve, with smaller models showing the largest absolute gains (Llama-3.1-8B: +35.8 pp on WorkArena; GPT-4o: +6.0 pp).
The contextualized observations are produced by the authors' LCOW module, so the improvement is conditional on that specific contextualization. The paper acknowledges that LCOW does not generalize to unseen UI element categories (0% on filter-list tasks for both GPT-4o and Gemini-1.5-Flash).