Self-Consistency / Self-consistency prompting / Wang et al. 2023 (self-consistency) / Wang et al. 2023b (Self-Consistency)

anchor

Findings

• IC-078GPT-4's self-verification loop causes performance collapse due to high false negative rates in binary verification [compared-to]
• IC-080GPT-4's performance is largely insensitive to the content of feedback; simple re-prompting with a sound verifier (sampling) matches or exceeds detailed critique [compared-to]
• IC-1157GPT-4 and other LMs show a large gap between rule induction and rule application, with task accuracy dropping to near zero on MiniScan when the LM itself applies its own proposed rules [compared-to]
• IC-1159GPT-4 is a strong inductive hypothesis proposer, achieving high accuracy on inductive reasoning benchmarks when its generated rules are applied by a symbolic interpreter [compared-to]
• IC-1168For GPT-4 Code Interpreter, code-based self-verification improves MATH accuracy to 73.54% while natural language self-verification slightly degrades it to 69.29% relative to the 69.69% base [compared-to]
• IC-1170GPT-3.5, Llama2, PaLM2, and GPT-4 are susceptible to a CoT-prompting backdoor attack (BadChain) on complex reasoning tasks, with stronger reasoning models showing higher attack success rates [builds-on]
• IC-1170GPT-3.5, Llama2, PaLM2, and GPT-4 are susceptible to a CoT-prompting backdoor attack (BadChain) on complex reasoning tasks, with stronger reasoning models showing higher attack success rates [primary]
• IC-522LLMs perform correct example inference without inducing the correct rule, and this gap is robust to prompting methods, fact count, and scenario form [compared-to]
• IC-700GPT-4's reasoning accuracy degrades when provided with incorrect hints from a weaker model [primary]
• IC-7951D subspaces of MLP activations found by DAS in GPT-2 Small (IOI) and GPT-2 XL (factual recall) produce apparent causal effects that are interpretability illusions driven by causally disconnected components activating dormant pathways [context]
• IC-840GPT-2 Small's name mover heads exhibit disrupted attention patterns under out-of-distribution Gaussian noise corruption [builds-on]
• IC-852Intrinsic self-correction without external feedback consistently degrades reasoning accuracy across GPT-3.5-turbo, GPT-4, GPT-4-turbo, and LLaMA-2-70B-chat [compared-to]
• IC-853Multi-agent debate with GPT-3.5-turbo-0301 does not outperform self-consistency at equivalent inference cost on GSM8K [compared-to]
• IC-858The choice of graph encoding method significantly changes LLM accuracy on graph reasoning tasks, with incident encoding outperforming adjacency by up to 34 percentage points on connected nodes [compared-to]