IC-1600Replacing a model's self-generated feedback with a stronger model's feedback consistently improves self-repair beyond both the i.i.d. baseline and the self-repair baseline
The paper tests the hypothesis that self-repair is bottlenecked by the model's inability to provide useful feedback on its own code. When CodeLlama's feedback is replaced by GPT-3.5 or GPT-4, and GPT-3.5's feedback is replaced by GPT-4, the boosted configurations beat both the corresponding i.i.d. baseline and the corresponding self-repair configuration at all sample budgets. On APPS, both CodeLlama and GPT-3.5 now beat their baselines, whereas in pure self-repair they mostly lag behind. On HumanEval, CodeLlama's gains increase with the strength of the feedback model.
Evidence
correlational
Key metric
In every boosted configuration (CodeLlama+GPT-3.5, CodeLlama+GPT-4, GPT-3.5+GPT-4), the boosted pass rate exceeds both the i.i.d. baseline and the self-repair baseline at all budgets. Repair success rates (Table 2): CodeLlama overall 1.1%, CodeLlama+GPT-3.5 2.2%, GPT-3.5 4.7%, GPT-3.5+GPT-4 11.5%, GPT-4 10.8%.
Caveat
The feedback model and repair model are different, so this tests cross-model transfer rather than the model's intrinsic repair ability. The experiment uses nf=10 instead of nf=25 to reduce cost.