IC-316Multi-turn code generation without CoT degrades performance for smaller Llama models and GPT-4o compared to single-turn repeated sampling under equal compute budgets
Kunhao Zheng, Juliette Decugis, Jonas Gehring, Taco Cohen, benjamin negrevergne, Gabriel Synnaeve
When the model is asked to retry after execution feedback but without any reasoning or instruction prompts, performance is at best flat and often worse than drawing independent single-turn samples. Llama 3.0 8B drops from 2.9 to 2.4 on pass 1@3, Llama 3.1 8B drops from 7.7 to 7.0, and GPT-4o drops from 27.6 to 23.1 on pass 1@3. The paper attributes this to the model's limited self-repair ability without explicit reasoning guidance: the model tends to make small modifications to its previous (incorrect) solution rather than attempting a fundamentally different approach.
GPT-4o result is at temperature 0.2 with small sampling budget; Llama results at temperature 1.0. The paper notes that multi-turn + CoT recovers and exceeds single-turn performance, so the degradation is specific to the absence of reasoning prompts.