On a balanced MMLU test set (equal numbers of questions with golden answers a, b, c, d), LLaMA-65B's zero-shot accuracy differs by answer choice: class 'a' achieves 71.58%, 'b' 59.65%, 'c' 58.95%, and 'd' 52.28%. With 5 demonstrations, the overall accuracy improves from 60.61% to 63.16%, but class 'a' accuracy drops to 54.74% while 'b', 'c', and 'd' all increase. This indicates a persistent preference for earlier answer choices in zero-shot that partially corrects with more demonstrations.
Measured on 1140 randomly sampled MMLU test cases with the test set randomly permuted to ensure balance across answer choices. The paper presents this as a confound to be controlled for in the spurious-connection experiments rather than as a primary contribution.