IC-1307Released LLMs (CodeLlama 7B/13B/34B, GPT-3.5, GPT-4) achieve limited code-optimization speedups with standard prompting, with the best baseline (GPT-3.5 CoT) reaching only 1.60x versus the 3.66x human reference
Alexander G Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R. Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, Amir Yazdanbakhsh
The paper evaluates five released LLMs on the PIE test set (978 C++ program pairs) using instruction-only, few-shot, and chain-of-thought prompting. Without task-specific training data, all models struggle: CodeLlama 7B achieves at most 1.13x speedup (CoT, best@8), CodeLlama 34B reaches 1.30x, GPT-3.5 reaches 1.60x, and GPT-4 reaches 1.58x. The human reference for the same test set is 3.66x average speedup. The gap between the best prompting baseline and human performance is roughly 2x.
Results are measured on a single test set of 978 competitive programming problems in C++; the 10% speedup threshold for counting a program as 'optimized' may undercount small but real improvements.