IC-073Released LLMs (GPT-4o, Llama-3.1-70B, Qwen2-7B, etc.) show limited workflow orchestration capability that degrades as workflow complexity increases

Shengda Fan, Xin Cong, Yuepeng Fu, Zhong Zhang, Shuyan Zhang, Yuanwei Liu, Yesai Wu, Yankai Lin, Zhiyuan Liu, Maosong Sun

SourceWorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models

The paper evaluates multiple released LLMs on the WorkflowBench test set under unseen-instruction (ID) and unseen-API (OOD) settings, measuring CodeBLEU and a GPT-4o-mini-based pass rate. All models achieve relatively low scores: GPT-4o reaches 28.5% CodeBLEU and 56.6% pass rate (ID), while Llama-3.1-70B reaches 27.3% and 55.4%. In-context learning improves scores modestly (GPT-4o to 30.2% CodeBLEU, 67.5% pass rate). A complexity breakdown (Figure 5) shows that every model's CodeBLEU score declines as the number of actions, branches/loops, or nested depth of the reference workflow increases, indicating a reproducible degradation under high-complexity conditions.

Evidence

correlational

Key metric

GPT-4o CodeBLEU 28.5% (ID) / 27.7% (OOD), pass rate 56.6% (ID) / 47.5% (OOD); w/ ICL 30.2% / 30.0% CodeBLEU, 67.5% / 57.6% pass rate. Llama-3.1-70B 27.3% / 27.2% CodeBLEU, 55.4% / 42.3% pass rate. Qwen2-7B 24.4% / 24.5% CodeBLEU, 25.6% / 22.6% pass rate.

Caveat

Evaluation is limited to static analysis (CodeBLEU and LLM-judged pass rate) without actual execution of the generated workflows. APIs are limited to Apple Shortcuts.

Model

GPT-4o mini, Llama 3.1 8B, 70B, Qwen 2 7B

Concepts

Failure mode

Datasets

WorkflowBench [eval]

Methods

CodeBLEU [eval], In-Context Learning / In-context learning prompt [supporting]

Related work

ProAgent [context], ToolLLM [context]

Related findings

IC-074, IC-075

Extraction

automatic-extraction