SourceWhat Makes Large Language Models Reason in (Multi-Turn) Code Generation?
The paper measures similarity between consecutive code attempts within a multi-turn trajectory. When given LDB-style feedback (debugger information with intermediate variable values), Llama 3.1 8B and 70B produce highly similar consecutive solutions (exploitative behavior), whereas binary or failed-tests feedback yields more diverse attempts (exploratory behavior). This reduced diversity translates to lower pass rates at large budgets: Llama 3.1 70B drops from 29.5 to 26.5 on pass 1@3 and from 56.2 to 54.8 on pass 100@300 when switching from failed-tests to LDB feedback. The paper posits that for hard competitive programming problems, the model's error is algorithmic (wrong approach) rather than a simple bug, so exploitation of a flawed solution is counterproductive.