The paper tests Llama-2-Chat (7B, 13B, 70B) on two human preference benchmarks (HHH Alignment and MT Bench Human Judgment) using a ranking grading scheme, where the model must assign a higher score to the human-preferred response. Despite no reward-model-specific training, Llama-2-Chat achieves 51.78%, 52.34%, and 53.67% total accuracy on HHH Alignment respectively. The authors attribute this to the RLHF training in Llama-2-Chat's base model. This is below GPT-4 (63.87%) and GPT-3.5-Turbo (57.12%) but above the dedicated StanfordNLP (44.79%) and ALMOST (49.90%) reward models.
Evidence
correlational
Key metric
HHH Alignment total accuracy: Llama-2-Chat 7B 51.78%, 13B 52.34%, 70B 53.67%; StanfordNLP 44.79%; ALMOST 49.90%; GPT-3.5-Turbo 57.12%; GPT-4 63.87%
Caveat
The ranking grading setting requires iterative inference at temperature 1.0 until a winner is obtained, which the authors note is 'not exactly fair compared to other ranking models.' The setting is described as a check of generalization, not a claim of SOTA.