Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Understanding Catastrophic Forgetting in Language Models via Implicit Inference
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-1053
Instruction tuning suppresses in-context learning in LLaMA, Vicuna, and OPT-IML, with the suppression being largest for English prompts and partially recoverable via translation to other languages
IC-1054
Code fine-tuning degrades English natural language reasoning in Code LLaMA relative to LLaMA-2, but the effect is negligible or slightly positive in French, Spanish, and German
IC-1055
Safety fine-tuning suppresses harmful content generation in ChatGPT relative to GPT-3.5, but the suppression is substantially weaker for non-English prompts