IC-1231On APPS, released code generation models span pass@1 from 0.20 (GPT-3 175B) to 6.20 (CodeRL), with value-based and policy-based RL methods outperforming supervised baselines
Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, Hongxia Yang
The paper reports pass@1, pass@5, and pass@1000 on the APPS benchmark for nine released code generation models across three difficulty levels (introductory, interview, competition). Supervised models range from GPT-2 0.1B (pass@1=1.00) to Codex 12B (pass@1=4.14) and AlphaCode 1B (pass@1000=17.67). RL-based methods CodeRL (pass@1=6.20), PPOCoder (pass@1=5.20), and RLTF (pass@1=4.16) generally outperform the supervised baselines at pass@1. The authors note that CodeRL's raw generation (pass@1=1.60, pass@5=3.28 with greedy/sampled decoding) is comparable to or exceeds B-Coder's raw generation, and that CodeRL's programs can be further improved by B-Coder's recovered reward ranking.
Baseline numbers for GPT-2, GPT-3, GPT-Neo, GPT-J, Codex, AlphaCode, CodeRL, PPOCoder, and RLTF are duplicated from their original papers (marked with dagger symbols in Table 1), not re-measured by the authors. The authors note that raw B-Coder performance without ranking is mixed compared to CodeRL (Table 8: CodeRL pass@5=3.28 vs B-Coder pass@5=2.88).