IC-1310Chain-of-thought prompting provides notable code-optimization gains only for larger models (CodeLlama 34B, GPT-3.5, GPT-4) but not for CodeLlama 7B or 13B, consistent with an emergent capability
Alexander G Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R. Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, Amir Yazdanbakhsh
Comparing CoT to few-shot prompting at best@8, CoT outperforms few-shot for CodeLlama 34B (19.63% vs 13.70% opt), GPT-3.5 (43.05% vs 29.04%), and GPT-4 (42.74% vs 36.40%), but underperforms few-shot for CodeLlama 7B (7.46% vs 9.51%) and CodeLlama 13B (11.15% vs 13.70%). The authors explicitly link this to the emergent-capability finding of Wei et al. (2022a), noting the improvement is 'notably only for the larger codellama (13b and 34b) and gpt-3.5 models.'
Evidence
correlational
Key metric
CoT vs few-shot best@8: CodeLlama 7B 7.46% vs 9.51%; CodeLlama 13B 11.15% vs 13.70%; CodeLlama 34B 19.63% vs 13.70%; GPT-3.5 43.05% vs 29.04%; GPT-4 42.74% vs 36.40%
Caveat
The CodeLlama 13B gap (11.15% vs 13.70%) is modest; the paper's claim of 'notable' improvement is a qualitative judgment. The comparison is within a single test set of 978 problems.