IC-1518Domain finetuning of LLaMA 2 7B, LLaMA 2 13B, and GPT-2 XL on PubMed causes topic and style priors to shift dramatically, accounting for the majority of the probability change, while factual knowledge learning contributes only a small fraction
The authors finetune three released language models on up to 1 million PubMed abstracts and probe the resulting models with controlled-variable text pairs (same content, different topic/style/factuals) generated by ChatGPT. Likelihood ratios show that the probability of the training topic (biomedical) and style (academic) increases sharply relative to alternatives, while the factual/counterfactual ratio changes at a much slower rate. An ablation on derived datasets confirms that topic and style adaptation contributes the main part of both the loss reduction on PubMed (learning) and the loss increase on C4 (forgetting), whereas factual learning, though steady, accounts for only a small portion. The same pattern generalises to the legal (Pile of Law) and customer-review (Amazon Reviews) domains.
Evidence
correlational
Key metric
Factual/counterfactual likelihood ratio changes at a significantly slower rate than topic and style ratios (Figures 2, 9, 10); ablation (Figure 4) shows topic+style adaptation is the main contributor to loss change; MMLU 0-shot drops from 42.6 (baseline) to 37.3 (LR 2e-5, 1M docs) while MMLU clinical anatomy rises from 46.7 to 48.1
Caveat
Analysis limited to at most 1 million documents (<1B tokens); the decomposition of text into topic, style, and factual is approximate and the boundary between content and style is not always clear; quantitative observations may reflect characteristics of the specific corpus used.