IC-075GPT-4o-mini and Qwen2.5-72B achieve low precision and recall when used as API retrievers for workflow orchestration

Shengda Fan, Xin Cong, Yuepeng Fu, Zhong Zhang, Shuyan Zhang, Yuanwei Liu, Yesai Wu, Yankai Lin, Zhiyuan Liu, Maosong Sun

SourceWorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models

In an appendix experiment, the paper uses GPT-4o-mini and Qwen2.5-72B with in-context learning to retrieve relevant APIs from a list given a task query. GPT-4o-mini achieves 42.5% precision and 36.4% recall; Qwen2.5-72B achieves 40.6% precision and 40.7% recall. Manual inspection of 50 random cases showed 84% of retrieved APIs still fulfilled the task, suggesting the low metrics overstate the practical failure rate.

Evidence

correlational

Key metric

GPT-4o-mini precision 42.5%, recall 36.4%; Qwen2.5-72B precision 40.6%, recall 40.7%

Caveat

The paper notes that manual inspection of 50 cases revealed 84% of retrieved APIs successfully fulfilled the tasks, indicating the precision/recall metrics may overstate the practical impact.

Model

GPT-4o mini, Qwen2.5 Qwen2.5-72B

Concepts

Failure mode, Method artefact

Datasets

WorkflowBench [source]

Methods

In-Context Learning / In-context learning prompt [supporting], MiniLM-L6-V2 [supporting]

Related findings

IC-073, IC-074

Extraction

automatic-extraction