IC-302Llama-3.1 models perform between unigram-inference and bigram-inference on Markov chain ICL, with performance improving monotonically with model scale
Core Francisco Park, Ekdeep Singh Lubana, Hidenori Tanaka
The authors feed sequences generated from a finite Markov chain into Llama-3.1 at three scales (8B, 70B, 405B) and measure the KL divergence between the model's predicted next-state distribution and the ground-truth transition matrix. Without being told the sequence is Markovian, all three models achieve a KL between the unigram-inference baseline and the bigram-inference baseline, showing they extract some transition structure from context. Performance improves with scale: 405B achieves the lowest KL, followed by 70B, then 8B.
Evidence
correlational
Caveat
The authors describe this experiment as motivated by curiosity and note it is a preliminary exploration; no specific KL values are printed in the text, only the qualitative ordering is stated.