In the limitations section, the paper reports HumanEval and GSM8K scores for Llama-2 and Code Llama at 7B and 13B, sourced from the Open LLM Leaderboard. Code Llama, which continued pre-training on code data, substantially outperforms the corresponding Llama-2 base on both tasks at both scales. The authors use this as secondary validation that pre-training on code data improves coding and mathematical reasoning capabilities, consistent with their main experiments on Pangu.
Data is sourced from the Open LLM Leaderboard rather than measured directly by the authors; the paper notes these are preliminary validations and calls for further verification on larger models.