IC-1380Pythia and OPT small models exhibit non-trivial performance on BigBench tasks that is invisible under beam search but revealed by extensive random sampling
Four small released models (Pythia-410m, Pythia-1.4b, OPT-350m, OPT-1.3b) are evaluated on two BigBench subtasks (emoji movie, date understanding) using beam search and random sampling with 1, 100, and 10,000 samples. Under beam search, all four models show near-zero pass rates. However, as the number of random samples increases, the fraction of passed instances rises substantially, indicating that the models possess non-trivial capability that conventional single-decode evaluation fails to detect. The authors conclude that the apparent zero performance is an artefact of insufficient measurement resolution rather than a true absence of capability.
Evidence
correlational
Caveat
The pilot experiment uses only two BigBench subtasks and four small models; the authors note that the passuntil strategy is designed for tasks where a random baseline achieves p(s)=0, and multiple-choice tasks with non-zero random baselines are excluded.