IC-689Subjective randomness generation and sharp ICL transitions emerge only in larger or reward-fine-tuned models, absent in earlier GPT-3 variants and smaller open-source models
Eric J Bigelow, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Tomer Ullman
Earlier OpenAI models (text-davinci-002, text-davinci-001, text-curie-001) do not show subjective randomness behavior: text-davinci-002 and -001 are heavily biased and uncontrollable, and text-curie-001 generates sequences with mean 0.50 regardless of the specified p(tails). Among open-source models, smaller variants (7B, 13B) and larger non-DPO models (Llama-2-70B, Tulu-2-70B, Mixtral-8x7B) show flat in-context learning curves in randomness judgment, while tulu-2-dpo-70b shows sharp s-shaped transitions similar to GPT-3.5+. This indicates the capability requires both sufficient scale (50+ billion parameters) and reward-based fine-tuning (RLHF or DPO).
Evidence
correlational
Key metric
text-curie-001 generates sequences with y = .50 regardless of p(tails); tulu-2-dpo-70b shows sharp s-shaped ICL curves while llama-2-7b-hf, llama-2-13b-hf, tulu-2-7b, tulu-2-13b, tulu-2-70b, and mixtral-8x7b models show flat curves; results replicated on 2x NVIDIA A100 80GB GPUs
Caveat
The paper notes that as of January 2024, text-davinci-003 is no longer publicly accessible, limiting replicability. The authors hypothesize but do not confirm that fine-tuning methods (instruction fine-tuning, SFT, RLHF) are the cause of the emergent capability. Open-source model results are limited to a subset of the full experimental suite.