The paper evaluates several released LLMs on the WebArena-Lite benchmark (165 test cases across five websites) using a simple prompt that defines feasible actions and provides examples. GPT-4-turbo achieves 17.6% average success rate, GPT-4o achieves 13.9%, LLaMA-3.1-70B-Instruct achieves 12.7%, LLaMA-3.1-8B-Instruct achieves 4.8%, and GLM-4-chat achieves 6.1%. The previously released AutoWebGLM agent achieves 18.2%. These results establish a significant gap between proprietary and open-source LLMs as web navigation agents.
Evidence
correlational
Key metric
GPT-4-turbo 17.6%, GPT-4o 13.9%, LLaMA-3.1-8B-Instruct 4.8%, LLaMA-3.1-70B-Instruct 12.7%, GLM-4-chat 6.1%, AutoWebGLM 18.2% average SR on WebArena-Lite
Caveat
Results use a single simple prompt; the paper notes that high-performing agents rely on 'meticulously crafted prompts' and proprietary APIs, so the simple-prompt setting may understate model capability. AWM+GPT-4-0613 (35.5%) and WebPilot+GPT-4o (37.2%) from prior work show that method scaffolding substantially changes the picture.