In zero-shot en-to-xx translation, Llama-2-13b frequently generates text unrelated to the translation task, such as 'I'm voting to close this question as off-topic because it is not about programming' when asked to translate English to Russian. This causes BLEU scores near zero for en-to-cs (0.87) and en-to-ru (0.59). The issue is substantially alleviated by using prompts in the target language (BLEU rises to 20.80 for en-to-cs) or by few-shot in-context learning, but both strategies still lag behind the authors' fine-tuning method by over 10 BLEU and COMET points.
The paper attributes the issue to the pre-training phase but does not provide a mechanistic explanation; the off-target behavior is observed in the zero-shot setting and is not quantified across all test sentences.