IC-315CoT prompting (reasoning + instruction) yields larger relative gains for larger LLMs and harder problems in competitive code generation, with the effect reversing for the most capable models

Kunhao Zheng, Juliette Decugis, Jonas Gehring, Taco Cohen, benjamin negrevergne, Gabriel Synnaeve

SourceWhat Makes Large Language Models Reason in (Multi-Turn) Code Generation?

The paper grid-searches 8 reasoning prompts and 6 instruction prompts across Llama 3.0/3.1 (8B, 70B, 405B) and GPT-4o on CodeContests and TACO. The best reasoning+instruction combination improves pass@100 by +5.0% for Llama 3.0 8B, +9.3% for Llama 3.0 70B, +3.3% for Llama 3.1 8B, and +5.2% for Llama 3.1 70B. On the TACO very-hard split, Llama 3.0 8B pass@100 nearly doubles (2.1% to 3.9%). However, for Llama 3.1 405B, CoT degrades pass@1 by 2.7 points, and for GPT-4o, multi-turn alone degrades pass 1@3 by 4.5 points, indicating the benefit reverses at the top of the scale.

Evidence

correlational

Key metric

pass@100 gains: Llama 3.0 8B +5.0% (12.3→17.3), Llama 3.0 70B +9.3% (23.8→33.1), Llama 3.1 8B +3.3% (22.8→26.1), Llama 3.1 70B +5.2% (48.9→54.1); TACO very-hard Llama 3.0 8B: 2.1%→3.9%; Llama 3.1 405B pass@1: 27.8→25.1 (−2.7); GPT-4o pass 1@3 multi-turn: 27.6→23.1 (−4.5)

Caveat

Llama 3.1 405B and GPT-4o evaluated only in small sampling regimes (pass@1, pass 1@3) due to compute constraints; no single prompt combination works best across all models and sample sizes.

Model

Llama 3 Llama 3.0 8B, Llama 3.0 70B, Llama 3.1 8B, 70B, 405B, GPT-4o

Concepts

Scale-dependent behaviour

Datasets

CodeContests [eval], TACO [eval]

Methods

Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [primary]

Related work

Chain-of-Thought prompting / Chain-of-Thought (CoT) prompting / CoT prompting / Few-shot Chain of Thought / Few-shot CoT / Chain-of-Thought (CoT-S) / CoT-bag prompting / Few-shot CoT prompting / Rationale prompting / Zero-shot CoT prompting / Wei et al. (2022) chain-of-thought prompting [builds-on]

Related findings

IC-316, IC-317

Extraction

automatic-extraction