Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-424
Structural in-context learning is transient in MultiBERTs and Pythia-1.4B, disappearing after early training
IC-425
Pretrained GPT-2 Large fails at structural in-context learning on unseen tokens in a syllogism task
IC-426
MultiBERTs exhibit a pushdown phenomenon where syntactic information migrates from later to earlier layers as training progresses