The paper evaluates 18 released LLMs on WorfBench, measuring both F1_chain (linear subtask ordering) and F1_graph (full DAG structure with parallel edges). Every model scores significantly lower on F1_graph than F1_chain. The largest average gap is 20.05% for GLM-4-9B, and the smallest is 15.01% for Llama-3.1-70B. GPT-4 achieves 67.32% F1_chain but only 52.47% F1_graph. Error analysis of GPT-4 identifies four recurring error types: granularity, explicitness, graph structure, and format, all attributed to insufficient environmental knowledge.
Evidence
correlational
Key metric
GPT-4: F1_chain 67.32%, F1_graph 52.47%; GLM-4-9B average gap 20.05%; Llama-3.1-70B average gap 15.01%; Claude-3.5: F1_chain 66.70%, F1_graph 52.53%
Caveat
The benchmark's gold workflows were synthesized by GPT-4, so the evaluation may partially reflect GPT-4's own generation biases. The paper acknowledges that some queries may have quality issues despite filtering.