Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
In-Context Learning Dynamics with Random Binary Sequences
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-687
GPT-3.5+ models exhibit a gambler's fallacy bias and generate low-complexity sequences when asked to produce random binary sequences
IC-688
GPT-3.5-turbo-instruct-0914 shows sharp phase transitions in in-context learning of simple formal languages, transitioning from random generation to deterministic pattern repetition as context length increases
IC-689
Subjective randomness generation and sharp ICL transitions emerge only in larger or reward-fine-tuned models, absent in earlier GPT-3 variants and smaller open-source models