Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Learning Performance-Improving Code Edits
2024-01-16
· ICLR 2024 spotlight ·
anchor
Findings
IC-1307
Released LLMs (CodeLlama 7B/13B/34B, GPT-3.5, GPT-4) achieve limited code-optimization speedups with standard prompting, with the best baseline (GPT-3.5 CoT) reaching only 1.60x versus the 3.66x human reference
IC-1308
Dynamic retrieval-based few-shot prompting substantially improves released LLMs' code optimization, with GPT-4-0613 reaching 76.07% optimization rate and 3.93x speedup (best@8), exceeding the 3.66x human reference
IC-1309
GPT-4-0613 exhibits reduced output diversity relative to GPT-3.5: it outperforms on best@1 but underperforms on best@8 under CoT prompting
IC-1310
Chain-of-thought prompting provides notable code-optimization gains only for larger models (CodeLlama 34B, GPT-3.5, GPT-4) but not for CodeLlama 7B or 13B, consistent with an emergent capability