IC-1415BLOOMZ generalizes instruction-following to programming languages (Go, Rust) absent from its instruction data, scoring above the random baseline
Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, Shayne Longpre
BLOOMZ, a 176B parameter model trained primarily on natural language, is evaluated on HumanEvalPack across six programming languages. Go and Rust are not contained in BLOOMZ's instruction data, yet the model performs substantially above the 0.0 random baseline for these languages across most tasks (e.g., 16.4% on Go HumanEvalFix, 8.4% on Go HumanEvalSynthesize, 5.7% on Rust HumanEvalFix). The authors interpret this as evidence that models can generalize instructions to programming languages seen only at pretraining, analogous to cross-lingual generalization in natural languages.