pass@k / pass n@k / pass@1 / pass@t

anchor

Findings

• IC-1413GPT-4 achieves near-saturation on Python code synthesis (86.6% pass@1) but scores significantly lower on code repair (47.8% avg) and code explanation (52.1% avg) across six languages [eval]
• IC-1414Pretrained code models Starcoder and CodeGeex2 score 0.0% on code explanation across all six languages because they generate code instead of natural language [eval]
• IC-1415BLOOMZ generalizes instruction-following to programming languages (Go, Rust) absent from its instruction data, scoring above the random baseline [eval]
• IC-1599Self-repair at equivalent compute budget provides only modest and inconsistent gains over i.i.d. sampling for CodeLlama-13B-Instruct, GPT-3.5, and GPT-4 on HumanEval and APPS [eval]
• IC-1600Replacing a model's self-generated feedback with a stronger model's feedback consistently improves self-repair beyond both the i.i.d. baseline and the self-repair baseline [eval]