Comparing base, SIFT, and RLHF variants within the CodeLlama and Llama-2 families, the paper finds that instruction fine-tuning and RLHF alignment generally degrade multi-turn tool-use performance. SIFT hurts CodeLlama-34b by 11.1% in absolute success rate (no feedback) and 15.4% (with feedback); RLHF hurts Llama-2-70b by 8.5% and 8.7% respectively. Notable exceptions are Vicuna-v1.5-7b and Lemur-v1-70b SIFT, where SIFT on multi-turn ShareGPT data improves performance.
Evidence
correlational
Key metric
SIFT hurts CodeLlama-34b by 11.1% (sr5: 28.2 base vs 17.1 SIFT) and 15.4% (sr5_feedback: 42.7 vs 27.3); RLHF hurts Llama-2-70b by 8.5% (sr5: 26.4 base vs 17.9 RLHF) and 8.7% (sr5_feedback: 35.3 vs 26.6)
Caveat
The authors note it is hard to conclude that RLHF in general hurts model performance, and leave exploring the role of RLHF in multi-turn interaction for future work. Vicuna-v1.5 and Lemur-v1 are exceptions where SIFT helps.