The paper evaluates GPT-4 and GPT-3.5 in both a generative setting (produce a response) and a discriminative setting (select the correct response from candidates) across 13 language benchmarks. In 10 of 13 datasets, at least one model outperforms humans in generation while underperforming in discrimination; in 7 of 13, both models show this pattern. The gap widens with task difficulty: on XSum, GPT-4 drops from 100% accuracy with easy negatives to 19% with hard negatives, while humans maintain consistently high accuracy.
Evidence
correlational
Key metric
for 10 of the 13 datasets, sub-hypothesis 1 is supported in at least one model; for 7 of the 13 datasets, supported in both models; in the xsum task, gpt4 achieves 100% accuracy when selecting the correct answer from easy negatives, but this drops to 19% when confronted with hard negatives
Caveat
Dataset/benchmark contamination is a potential limitation with proprietary models; the authors note this should affect generation and discriminative evaluation similarly. The study focuses on a small set of the most popular models.