IC-1053Instruction tuning suppresses in-context learning in LLaMA, Vicuna, and OPT-IML, with the suppression being largest for English prompts and partially recoverable via translation to other languages
The paper constructs a controlled prompt where in-context exemplars define a latent task (repeat or capitalize) but the final query is phrased as an instruction (math or fill-in-the-blank). Pretrained models (LLaMA-7B, OPT-1.3B) perform the ICL task, while their instruction-tuned counterparts (Alpaca, Vicuna-7B, OPT-IML-1.3B) follow the instruction instead. The drop in ICL accuracy is largest in English (56.75% for Alpaca, 40.00% for Vicuna, 21.00% for OPT-IML) and much smaller in French, Spanish, Dutch, Hungarian, leetspeak, or Pig Latin. For example, Alpaca's ICL drop under leetspeak is only 1.0% versus 56.75% in English.
Translation is implemented via Google Translate and may introduce errors; the paper notes that translation can violate invertibility for tasks requiring contextual knowledge that varies across languages. Evaluation is limited to first-word completion accuracy over 400 samples and 4 ICL vs IF task combinations.