Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Amber-7B
text · generative ·
anchor
Findings
IC-185
In Pythia-1B and Amber-7B, the probability of memorizing a training sequence scales log-linearly with both the number of repetitions in the corpus and the z-complexity of the sequence
IC-186
The memorization status of sequences in Pythia-1B and Amber-7B is stationary throughout training: KL-LD fluctuations are mean-reverting with fixed variance, rejecting a random-walk model with p < 10⁻⁸
IC-187
Latent memorized sequences in Pythia-1B and Amber-7B can be recovered by adding random Gaussian noise of magnitude 2×10⁻³ to model weights, while un-memorized and unseen sequences cannot