IC-734GPT-3.5-turbo's factual verification F1 decreases as the number of reasoning hops required to validate a claim increases

Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufei Guo, Lijie Wen, Philip S. Yu, Zhijiang Guo

SourceTowards Understanding Factual Knowledge of Large Language Models

The paper categorizes multifaceted and structural factual questions by the number of hops (1, 2, 3, or multi-hop) needed to verify them, sampling 1,490 data pieces from each dataset. For GPT-3.5-turbo under few-shot CoT, macro F1 drops steadily: from 51.3 at 1-hop to 45.2 at multi-hop for multifaceted questions, and from 40.6 at 1-hop to 30.2 at 3-hop for structural questions. The authors attribute this to the extended reasoning chain requiring heightened logical reasoning capabilities.

Evidence

correlational

Key metric

multifaceted macro F1: 51.3 (1-hop), 48.4 (2-hop), 47.0 (3-hop), 45.2 (multi-hop); structural macro F1: 40.6 (1-hop), 39.4 (2-hop), 30.2 (3-hop)

Caveat

Analysis is reported only for GPT-3.5-turbo; the 1,490-sample subset was randomly drawn from the full dataset.

Model

GPT-3.5 / ChatGPT-3.5 GPT-3.5-turbo

Concepts

Failure mode

Methods

Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [eval]

Related findings

IC-733, IC-735, IC-736

Extraction

automatic-extraction