Findings
- IC-1440GCN, GAT, GraphSAGE, and SGC exhibit structure-dependent generalization in transductive node classification: test nodes with shorter paths to training nodes are classified more accurately [eval]
- IC-1518Domain finetuning of LLaMA 2 7B, LLaMA 2 13B, and GPT-2 XL on PubMed causes topic and style priors to shift dramatically, accounting for the majority of the probability change, while factual knowledge learning contributes only a small fraction [train]
- IC-1519Topic and style biases in LLaMA 2 7B are learned like simple features (rapidly, with minimal capacity, concentrated at the first few tokens, magnified by learning rate) while factual knowledge is learned like complex features (slowly, requiring significant capacity, uniformly across positions, unaffected by learning rate) [train]
- IC-370Knowledge entropy (sparsity of FFN memory coefficients) decreases consistently during pretraining for OLMo 1B, 7B, and Pythia 1.4B, and this decrease strongly correlates with reduced knowledge acquisition and increased forgetting in continual learning [train]
- IC-371Artificially resuscitating inactive memory vectors by scaling the up-projection matrix K improves knowledge acquisition and reduces forgetting, with the effect more pronounced for later-stage OLMo models [train]