Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting

anchor

Prompt the model to produce intermediate reasoning steps before its answer.

Findings

• IC-017Truncating MLP weights improves few-shot Chain-of-Thought reasoning accuracy on GSM8K for Phi-3 and Llama-3.1-8B [primary]
• IC-097Chain-of-thought prompting outperforms few-shot direct QA for Llama 3 models above 8B parameters, while few-shot is best below 3B [primary]
• IC-101GPT-4 and Claude 3 struggle to emulate a lower capability profile (high school freshman level) via zero-shot prompting, with only moderate improvement from chain-of-thought prompting [compared-to]
• IC-109GPT-3.5-turbo and GPT-4 produce cycles in inferred causal graphs when using pairwise prompts, with cycle counts growing sharply on larger graphs [compared-to]
• IC-1152Token reinforcement in demonstrations constrains LLaMA-65B's output to valid label spaces on MMLU and enables chain-of-thought pattern following on GSM8K without requiring question content [context]
• IC-1166MPLUG-Owl's VQA accuracy on VQA-X increases from 68.30% to 74.48% when prompted with progressively higher-quality rationales generated by RAPPER [primary]
• IC-1170GPT-3.5, Llama2, PaLM2, and GPT-4 are susceptible to a CoT-prompting backdoor attack (BadChain) on complex reasoning tasks, with stronger reasoning models showing higher attack success rates [builds-on]
• IC-1170GPT-3.5, Llama2, PaLM2, and GPT-4 are susceptible to a CoT-prompting backdoor attack (BadChain) on complex reasoning tasks, with stronger reasoning models showing higher attack success rates [primary]
• IC-1270Chain-of-thought prompting does not mitigate privacy leakage in GPT-4 or ChatGPT [builds-on]
• IC-1270Chain-of-thought prompting does not mitigate privacy leakage in GPT-4 or ChatGPT [primary]
• IC-1307Released LLMs (CodeLlama 7B/13B/34B, GPT-3.5, GPT-4) achieve limited code-optimization speedups with standard prompting, with the best baseline (GPT-3.5 CoT) reaching only 1.60x versus the 3.66x human reference [primary]
• IC-1309GPT-4-0613 exhibits reduced output diversity relative to GPT-3.5: it outperforms on best@1 but underperforms on best@8 under CoT prompting [primary]
• IC-1310Chain-of-thought prompting provides notable code-optimization gains only for larger models (CodeLlama 34B, GPT-3.5, GPT-4) but not for CodeLlama 7B or 13B, consistent with an emergent capability [primary]
• IC-1327GPT-4, GPT-3.5, Llama2, and Vicuna models underperform human annotators on multistep soft reasoning in natural language narratives, with smaller models scoring near random chance [builds-on]
• IC-1327GPT-4, GPT-3.5, Llama2, and Vicuna models underperform human annotators on multistep soft reasoning in natural language narratives, with smaller models scoring near random chance [compared-to]
• IC-1499Self-rationalization quality and task accuracy scale with model size across GPT-3, FLAN-T5, and LLaMA on five QA datasets [eval]
• IC-174RAG reduces model abstention and LLMs hallucinate rather than abstain when the retrieved context is insufficient to answer the query [supporting]
• IC-204GPT-4o performs worse with explicit chain-of-thought prompting than with the original prompt on OpenRCA tasks [builds-on]
• IC-204GPT-4o performs worse with explicit chain-of-thought prompting than with the original prompt on OpenRCA tasks [compared-to]
• IC-221GPT-4o ReAct success rate drops from 47% on synchronous to 11% on asynchronous planning tasks, and all other tested LLMs show equal or worse performance [builds-on]
• IC-264All 18 evaluated LLMs fail to abstain when the provided context lacks the answer, with performance gaps of 13.6% to 68.4% relative to the original context [compared-to]
• IC-265Model families show extreme variation in detecting conflicting answers in inconsistent contexts, with phi-3 series at 5.8% average accuracy versus GPT-4 series at 89.35% [compared-to]
• IC-276All 14 evaluated VLMs show a large gap between average-case and worst-case accuracy on DynaMath variants, with worst-case at or below 50% of average-case, and the failures are systematic rather than random [compared-to]
• IC-309As zero-shot safety judges, GPT-4o achieves 78.9% Cohen's kappa agreement with human annotators while Llama-3-8B-instruct (39.0%) and Mistral-7B-instruct-v0.2 (53.9%) perform substantially worse [compared-to]
• IC-315CoT prompting (reasoning + instruction) yields larger relative gains for larger LLMs and harder problems in competitive code generation, with the effect reversing for the most capable models [builds-on]
• IC-315CoT prompting (reasoning + instruction) yields larger relative gains for larger LLMs and harder problems in competitive code generation, with the effect reversing for the most capable models [primary]
• IC-333Llama-3-405B overexplains math solutions with structured markdown headings and conversational tone compared to GPT-4o's concise formal notation, achieving 97% model-matching accuracy [supporting]
• IC-362LLMs with chain-of-thought prompting predict and simulate human risky choices that are more rational than actual human behavior, correlating more highly with maximum expected value than with human choices [primary]
• IC-363LLM inferences about others' preferences from observed decisions are highly correlated with human inferences because both assume the decision-maker is rational [primary]
• IC-474GPT-4, GPT-4o, and Llama-3.1-405B fail at knowledge classification and comparison without chain-of-thought [compared-to]
• IC-474GPT-4, GPT-4o, and Llama-3.1-405B fail at knowledge classification and comparison without chain-of-thought [context]
• IC-487Including sub-questions in the prompt improves LLM performance on multi-hop QA tasks [primary]
• IC-492Least-to-most prompting consistently improves MLLM accuracy on the relationship application step compared to direct answering, with GPT-4o improving from 0.9% to 6.44% on VOILA-WD [compared-to]
• IC-555Large LLMs (GPT-3.5-turbo, Gemini 1.5 Flash, Llama3-70B, Mixtral 46.7B) exhibit reasoning errors and significant accuracy degradation on large-scale logical commonsense reasoning tasks with 32k+ rules, even when the knowledge base is complete and retrieval is ideal [primary]
• IC-586Symbolic distance (number of reasoning steps) is the primary bottleneck for relational reasoning in LLMs, not total context length [primary]
• IC-600Chain-of-thought prompting improves most LMMs on synthetic detection but degrades LLaVA-ov-7b from 56.6% to 18.8%, while GPT-4o performs well without it (64.1% baseline) [primary]
• IC-733CoT prompting improves factual accuracy for instruction-tuned LLMs but degrades it for non-instruction-tuned LLMs such as OPT, BLOOM, and LLaMA [eval]
• IC-734GPT-3.5-turbo's factual verification F1 decreases as the number of reasoning hops required to validate a claim increases [eval]
• IC-735GPT-3.5-turbo's factual verification performance drops substantially under adversarial modifications, with man-made adversarial examples causing the largest decline [eval]
• IC-736Vicuna-13B outperforms Vicuna-7B on factual knowledge tasks by 5.4% on average [eval]
• IC-858The choice of graph encoding method significantly changes LLM accuracy on graph reasoning tasks, with incident encoding outperforming adjacency by up to 34 percentage points on connected nodes [primary]