Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Understanding In-Context Learning from Repetitions
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-1151
LLaMA, OPT, LLaMA-2, Mistral, and GPT-J all exhibit token co-occurrence reinforcement, where the probability of generating a token increases monotonically with the number of its contextual co-occurrences
IC-1152
Token reinforcement in demonstrations constrains LLaMA-65B's output to valid label spaces on MMLU and enables chain-of-thought pattern following on GSM8K without requiring question content
IC-1153
Intentionally constructed spurious token connections in MMLU demonstrations misdirect LLaMA-65B's in-context learning toward specific answer choices
IC-1154
LLaMA-65B exhibits a selection bias where zero-shot accuracy varies substantially by answer choice, with 'a' at 71.58% and 'd' at 52.28%