SourceKnowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition
To test whether low knowledge entropy causally limits acquisition, the authors modify the up-projection matrix K of the final-stage OLMo checkpoint: they identify the lowest p% of memory coefficients and multiply the corresponding rows of K by a factor u that scales them toward the layer mean, controlled by an amplifying factor q. With p fixed at 50, setting q to 1, 2, or 3 generally improves both acquisition and retention over the unmodified model, while q=0.5 (further suppressing inactive coefficients) worsens both. The benefit grows as the original model is from a later pretraining stage. However, even after resuscitation, performance remains below that of a mid-stage model with equivalent knowledge entropy, suggesting linear scaling of a subset of parameters is insufficient to fully restore plasticity.