Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
AlphaCode
anchor
Findings
IC-1231
On APPS, released code generation models span pass@1 from 0.20 (GPT-3 175B) to 6.20 (CodeRL), with value-based and policy-based RL methods outperforming supervised baselines
IC-1307
Released LLMs (CodeLlama 7B/13B/34B, GPT-3.5, GPT-4) achieve limited code-optimization speedups with standard prompting, with the best baseline (GPT-3.5 CoT) reaching only 1.60x versus the 3.66x human reference
[context]