IC-1193Vicuna and Alpaca achieve 0% pass rate on all ToolBench tool-use instructions, while GPT-4 and ChatGPT reach 71.1% and 64.8% with DFSDT, revealing a wide capability gap in tool use among released LLMs
The paper evaluates six released LLMs on ToolBench, a benchmark of 126,486 tool-use instructions spanning single-tool and multi-tool scenarios with 16,464 real-world REST APIs. Vicuna and Alpaca, both fine-tuned for general instruction following, fail to pass any instruction across all six evaluation splits (i1-inst, i1-tool, i1-cat, i2-inst, i2-cat, i3-inst), yielding 0.0 pass rate and 0.0 win rate. In contrast, GPT-4 with DFSDT achieves 71.1% average pass rate and ChatGPT with DFSDT achieves 64.8%, while Claude-2 (22.6%) and Text-Davinci-003 (43.1%) fall in between. The authors conclude that current instruction tuning largely focuses on language skills and does not cover the tool-use domain.
All models except ToolLLaMA-dfsdt-retriever use the oracle (ground-truth) API set rather than a retriever, so results reflect reasoning ability given the correct tools rather than end-to-end tool selection. The evaluation uses ToolEval (ChatGPT-based automatic evaluator) with 87.1% pass-rate and 80.3% win-rate agreement with human annotators.