SourceUncovering Latent Memories in Large Language Models
The paper measures KL-Levenshtein distance (KL-LD) between the model's greedy continuation and the true continuation for sequences of varying repetition count and z-complexity (ratio of compressed to original length). For both Pythia-1B and Amber-7B, sequences with more repetitions and lower z-complexity are more easily memorized, and the relationship is log-linear in both factors. Lower-complexity strings require fewer repeats to reach the same memorization level.