IC-1327GPT-4, GPT-3.5, Llama2, and Vicuna models underperform human annotators on multistep soft reasoning in natural language narratives, with smaller models scoring near random chance
The paper evaluates seven released LLMs on 756 multistep soft reasoning examples across three domains (murder mystery, object placements, team allocation) using a CoT+ prompting strategy. GPT-4 achieves the highest accuracy at 80.4/60.9/68.4% across the three domains, but still falls well short of the human majority-vote baseline of 94.1/95.0/100.0%. GPT-3.5-turbo scores 61.6/46.9/40.4%, while Llama2 (70B and 7B) and all Vicuna variants (7B, 13B, 33B) score at or near random chance (50.0/24.6/33.3%). Varying the prompting strategy from regular to CoT to CoT+ to 1-shot/few-shot CoT+ provides limited improvement for GPT-4, with the best result (few-shot CoT+ on team allocation: 89.6%) still below human performance.
The dataset was generated using GPT-4, so GPT-4 may have some familiarity with the narrative style, though the authors argue the hidden reasoning structure prevents trivial solvability. The human evaluation used only 7 annotators (4 of whom are paper authors).