The paper tests whether providing a workflow (generated by the authors' fine-tuned Qwen-2-7B) as prior knowledge improves end-to-end task performance. GPT-4 gains +13.57% on ALFWorld (27.14 to 40.71) and +18.65% on WebShop (28.36 to 47.01). Llama-3.1-8B gains +6.72% on ALFWorld and +7.14% on WebShop. Qwen-2-72B gains +2.86% on ALFWorld and +5.57% on WebShop. The gains are larger for weaker models and on ALFWorld (more diverse environments) than WebShop. Average planning steps also decrease with workflow knowledge.
Evidence
correlational
Key metric
GPT-4: ALFWorld 27.14 to 40.71 (+13.57), WebShop 28.36 to 47.01 (+18.65); Llama-3.1-8B: ALFWorld 1.49 to 8.21 (+6.72), WebShop 5.00 to 12.14 (+7.14); Qwen-2-72B: ALFWorld 53.57 to 56.43 (+2.86), WebShop 56.72 to 62.29 (+5.57)
Caveat
The workflow is generated by the authors' fine-tuned 7B model, so the quality of the prior knowledge is tied to that specific model. The paper notes the workflow is generated by a 7B model providing guidance to a 72B model, raising questions about the 'weak-guide-strong' paradigm.