Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Is Self-Repair a Silver Bullet for Code Generation?
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-1599
Self-repair at equivalent compute budget provides only modest and inconsistent gains over i.i.d. sampling for CodeLlama-13B-Instruct, GPT-3.5, and GPT-4 on HumanEval and APPS
IC-1600
Replacing a model's self-generated feedback with a stronger model's feedback consistently improves self-repair beyond both the i.i.d. baseline and the self-repair baseline
IC-1601
GPT-4's self-generated feedback is significantly less effective than human programmer feedback for code repair, with the gap widening on harder problems