IC-1380Pythia and OPT small models exhibit non-trivial performance on BigBench tasks that is invisible under beam search but revealed by extensive random sampling

Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, Maosong Sun

SourcePredicting Emergent Abilities with Infinite Resolution Evaluation

Four small released models (Pythia-410m, Pythia-1.4b, OPT-350m, OPT-1.3b) are evaluated on two BigBench subtasks (emoji movie, date understanding) using beam search and random sampling with 1, 100, and 10,000 samples. Under beam search, all four models show near-zero pass rates. However, as the number of random samples increases, the fraction of passed instances rises substantially, indicating that the models possess non-trivial capability that conventional single-decode evaluation fails to detect. The authors conclude that the apparent zero performance is an artefact of insufficient measurement resolution rather than a true absence of capability.

Evidence

correlational

Caveat

The pilot experiment uses only two BigBench subtasks and four small models; the authors note that the passuntil strategy is designed for tasks where a random baseline achieves p(s)=0, and multiple-choice tasks with non-zero random baselines are excluded.

Model

Pythia, OPT

Concepts

Method artefact

Datasets

Big-Bench / Big-Bench verb tense [eval]

Related work

Schaeffer et al. 2023 (Are emergent abilities of large language models a mirage?) [context]

Extraction

automatic-extraction