IC-1169CodeLlama-7B and CodeLlama-34B show improvement with CSV prompting on GSM8K and MATH but at much lower absolute accuracy than GPT-4 Code Interpreter

Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, Hongsheng Li

SourceSolving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification

In Appendix C, the authors apply the same CSV zero-shot prompt to CodeLlama-7B and CodeLlama-34B. On GSM8K, CodeLlama-7B improves from 17.44% to 20.85% and CodeLlama-34B from 28.96% to 37.60%. On MATH, CodeLlama-7B improves from 6.56% to 10.18% and CodeLlama-34B from 9.12% to 13.36%. The absolute accuracy remains far below GPT-4 Code Interpreter's 92.9% (GSM8K) and 69.69% (MATH), indicating that the base model's code generation and execution capabilities are critical for the method to be effective.

Evidence

correlational

Key metric

GSM8K: CodeLlama-7B 17.44%→20.85%, CodeLlama-34B 28.96%→37.60%; MATH: CodeLlama-7B 6.56%→10.18%, CodeLlama-34B 9.12%→13.36%

Caveat

Results are from a single zero-shot prompt with no voting; the paper does not test VW-voting on CodeLlama. The absolute accuracy gap suggests the method's effectiveness is heavily dependent on the base model's code capabilities.

Model

CodeLlama-13B CodeLlama-7B, CodeLlama-34B

Datasets

GSM8K [eval], MATH [eval]

Related findings

IC-1167, IC-1168

Extraction

automatic-extraction