IC-439Agent-specialized fine-tuned models (XLAM) significantly improve API selection over base models, but code-fine-tuned models (AgentLM) degrade performance, and no fine-tuning approach improves input recognition
Haiyang SHEN, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, Yun Ma
The paper evaluates 6 fine-tuned agent models (AgentLM-7B/13B/70B, XLAM-7B-R/8x7B-R, Lemur-70B-Chat-V1) against their base models (Llama-2-7B/13B/70B, Mistral-7B, Mixtral-8x7B) on ShortcutsBench. XLAM-7B-R achieves 70.56% API selection accuracy versus 17.03% for its base Mistral-7B, a large improvement. However, AgentLM-7B scores 24.95% versus 32.75% for Llama-2-7B, showing degradation. For parameter filling, XLAM shows improvement but accuracy drops sharply with context length. For input recognition, none of the fine-tuned models show any improvement over their base models, with most scoring 0-12%.
Evidence
correlational
Key metric
XLAM-7B-R API selection 70.56% vs Mistral-7B base 17.03%; AgentLM-7B API selection 24.95% vs Llama-2-7B base 32.75%; XLAM-8x7B-R API selection 85.29% vs Mixtral-8x7B base 70.21%; input recognition for all fine-tuned models at or below 12%
Caveat
Context length limited to 4k tokens for most models and 32k for XLAM series; results reported only for first 3 difficulty levels; outputs processed using GPT-4o-mini for format conversion