IC-1157GPT-4 and other LMs show a large gap between rule induction and rule application, with task accuracy dropping to near zero on MiniScan when the LM itself applies its own proposed rules

Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, Xiang Ren

SourcePhenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement

The paper tests whether LMs can apply the rules they themselves proposed. When a symbolic interpreter applies the LM-induced rules, performance is strong (e.g., 85.0% task accuracy on MiniScan for GPT-4). When the LM is instead asked to apply its own rules, performance collapses: MiniScan task accuracy drops to 0.0%, ACRE from 59.0% to 47.0%, List Functions from 61.2% to 50.0%, and MiniARC from 14.6% to 5.4%. The same pattern holds for GPT-3.5, Claude-2, and Llama2-70B. More advanced prompting (self-consistency, zero-shot CoT) does not close the gap.

Evidence
correlational
Key metric
GPT-4 task accuracy: MiniScan 85.0% (symbolic) vs 0.0% (LM); ACRE 59.0% vs 47.0%; List Functions 61.2% vs 50.0%; MiniARC 14.6% vs 5.4%. GPT-3.5 MiniScan 8.0% vs 0.0%. Claude-2 MiniScan 13.0% vs 0.0%. Llama2-70B MiniScan 0.0% vs 0.0%.
Caveat
The authors note that prior work has shown LMs fall short on symbolic tasks, and they do not claim the LM interpreter should match the symbolic interpreter. However, the gaps are described as 'so large' that they are 'strong indicators of LMs' puzzling behaviors.'
Model
GPT-3.5 / ChatGPT-3.5, Llama 2 / Llama 2 base Llama 2 70B, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report GPT-4V / GPT-4 vision
Concepts
Failure mode
Datasets
ACRE [eval], MiniScan [eval], MiniARC [eval]
Methods
Hypothesis Refinement / Iterative hypothesis refinement [primary], Self-Consistency / Self-consistency prompting / Wang et al. 2023 (self-consistency) / Wang et al. 2023b (Self-Consistency) [compared-to], Zero-shot Chain-of-Thought / Wei et al. 2022 (Chain of Thought) / Zero-shot chain-of-thought prompting [compared-to]
Related work
Hypothesis search: inductive reasoning with language models [context]
Related findings
IC-1158, IC-1159
Extraction
automatic-extraction