SourceOctoPack: Instruction Tuning Code Large Language Models
When evaluated on the HumanEvalExplain task, which requires generating a natural language description of a code function, both Starcoder and CodeGeex2 produce code output rather than natural language explanations. The authors manually inspect the first ten Python samples and disqualify a model if none of the outputs are explanations; both models are disqualified. The authors attribute this to these models being only conditioned to write code during pretraining, without exposure to natural language targets during instruction tuning.