Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
What Makes Large Language Models Reason in (Multi-Turn) Code Generation?
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-315
CoT prompting (reasoning + instruction) yields larger relative gains for larger LLMs and harder problems in competitive code generation, with the effect reversing for the most capable models
IC-316
Multi-turn code generation without CoT degrades performance for smaller Llama models and GPT-4o compared to single-turn repeated sampling under equal compute budgets
IC-317
More detailed execution feedback (LDB) induces exploitative behavior in Llama 3.1 models, reducing code diversity and hurting performance at large sample budgets