The paper evaluates 10 LLMs under four prompting strategies (zero-shot, zero-shot CoT, few-shot, few-shot CoT) on the Pinocchio benchmark. Instruction-tuned models benefit from CoT with an average 2.1% improvement, while non-instruction-tuned models (OPT, BLOOM, LLaMA) show stable or decreased accuracy. The authors attribute this to untuned models producing verbose content considerations under CoT that overshadow the factual discernment task, leading to incorrect label outputs. Overall, LLMs without instruction tuning underperform those with instruction tuning by 16.0%.
Evidence
correlational
Key metric
instruction-tuned vs non-instruction-tuned gap: 16.0%; few-shot over zero-shot: 7.3% average improvement; CoT for tuned models: +2.1% average; OPT-6.7B few-shot acc 36.9 vs zero-shot (no output); BLOOM-7B zero-shot acc 29.7 vs zero-shot CoT acc 14.8; LLaMA-7B zero-shot acc 31.8 vs zero-shot CoT acc 22.3
Caveat
The paper notes that due to limited computing resources, hyperparameter exploration was only done for Vicuna-7B and Vicuna-13B. OPT-6.7B produced no valid outputs under zero-shot settings, so its zero-shot numbers are unavailable.