IC-370Knowledge entropy (sparsity of FFN memory coefficients) decreases consistently during pretraining for OLMo 1B, 7B, and Pythia 1.4B, and this decrease strongly correlates with reduced knowledge acquisition and increased forgetting in continual learning
Jiyeon Kim, Hyunji Lee, Hyowon Cho, Joel Jang, Hyeonbin Hwang, Seungpil Won, Youbin Ahn, Dohaeng Lee, Minjoon Seo
The paper measures how broadly a model activates its feed-forward memory vectors (the rows of the second projection matrix V) by computing the entropy of the average memory coefficient distribution across all tokens. For OLMo 1B, 7B, and Pythia 1.4B, this knowledge entropy declines monotonically as pretraining progresses, with the sharpest drop in the last layer. In a continual knowledge learning setup (further training on PubMed or C4 with injected fictional knowledge), models from later pretraining stages show lower acquisition rates and higher forgetting rates. The Pearson correlation between knowledge entropy and acquisition is 0.94 (p=6e-5) and with forgetting is -0.96 (p=1e-5). Mid-stage checkpoints (~50% of the learning rate schedule) achieve the best balance of acquisition, retention, and overall performance.
Evidence
correlational
Key metric
Pearson correlation between knowledge entropy and acquisition: 0.94 (p=6e-5); with forgetting: -0.96 (p=1e-5). Baseline acquisition a(θ) across OLMo 1B pretraining steps 118k/369k/554k/738k: 25.2/24.8/21.4/18.8%; forgetting f(θ): 10.5/12.9/17.0/19.5%. Downstream performance p(θ): 54.0/54.6/52.8/52.0%.
Caveat
The study is limited to OLMo 1B and 7B (and Pythia 1.4B in the appendix) because they are the only models that publicly provide intermediate pretraining checkpoints. The authors note that extending to other models is future work. The mid-point is approximated as 50% of the learning rate schedule, and defining it precisely remains an open question.