Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement
2024-01-16
· ICLR 2024 oral ·
anchor
Findings
IC-1157
GPT-4 and other LMs show a large gap between rule induction and rule application, with task accuracy dropping to near zero on MiniScan when the LM itself applies its own proposed rules
IC-1158
GPT-4 and other LMs are brittle to noisy exemplars and unfamiliar output representations, with performance degrading sharply even under minimal perturbation
IC-1159
GPT-4 is a strong inductive hypothesis proposer, achieving high accuracy on inductive reasoning benchmarks when its generated rules are applied by a symbolic interpreter