On a 1k-sample human-annotated test set where three annotators independently labeled which of two LLM responses is better (with IAA > 0.85), GPT-3.5 achieves accuracy 0.6296, precision 0.6195, recall 0.6359, F1 0.5820, and GPT-4 achieves accuracy 0.6647, precision 0.6620, recall 0.6815, F1 0.6180. Both models use a prompt-based approach with response order swapped and conflicting results revised to tie. The test set covers diverse tasks sampled from the Self-Instruct human evaluation dataset.
The evaluation is on a single 1k-sample test set; the paper notes that the core training data for PandaLM is derived from GPT-3.5, which may introduce distributional alignment between GPT-3.5's evaluation style and the test set.