SourceOctoPack: Instruction Tuning Code Large Language Models
BLOOMZ, a 176B parameter model trained primarily on natural language, is evaluated on HumanEvalPack across six programming languages. Go and Rust are not contained in BLOOMZ's instruction data, yet the model performs substantially above the 0.0 random baseline for these languages across most tasks (e.g., 16.4% on Go HumanEvalFix, 8.4% on Go HumanEvalSynthesize, 5.7% on Rust HumanEvalFix). The authors interpret this as evidence that models can generalize instructions to programming languages seen only at pretraining, analogous to cross-lingual generalization in natural languages.