IC-187Latent memorized sequences in Pythia-1B and Amber-7B can be recovered by adding random Gaussian noise of magnitude 2×10⁻³ to model weights, while un-memorized and unseen sequences cannot

Sunny Duan, Mikail Khona, Abhiram Iyer, Rylan Schaeffer, Ila R Fiete

SourceUncovering Latent Memories in Large Language Models

For sequences that were un-memorized (KL-LD > 50) at checkpoint 10k but memorized (KL-LD < 10) at a later checkpoint, the authors add Gaussian noise (magnitude 2×10⁻³) to each weight parameter and repeat 200 times, selecting the perturbation yielding the lowest KL-LD. This recovers the latent memories, and the resulting distribution matches that of the later checkpoint. In contrast, sequences never memorized and sequences not yet seen by the model cannot be recovered by perturbation. As a control, sampling at four different temperatures (200 samples each) fails to recover the latent memories. Additionally, latent memorized sequences exhibit significantly lower cross-entropy loss than un-memorized or unseen sequences, providing a diagnostic.

Evidence

interventional

Key metric

Gaussian noise magnitude 2 × 10⁻³, 200 perturbations; latent sequences un-memorized at 10k (KL-LD > 50) but memorized at 19k (KL-LD < 10) are recovered; sampling at 4 temperatures with 200 samples each fails to recover

Caveat

The perturbation magnitude (2×10⁻³) was chosen by the authors; the paper notes further testing across other frontier models is required and the mechanistic explanation for why training acts as random noise with respect to memorized sequences requires further study.

Model

Pythia 1B, Amber-7B

Datasets

The Pile [source]

Related findings

IC-185, IC-186

Extraction

automatic-extraction