SourceThe Trickle-down Impact of Reward Inconsistency on RLHF
The paper evaluates the released StackLLama checkpoint on contrast instructions, a benchmark of lexically similar instruction pairs with different ground-truth responses. A consistent reward model should assign higher reward to the correct instruction-response pairing than to the swapped distractor. StackLLama scores 50.1 on response consistency (cres) and 48.2 on instruction consistency (cins) for the Stack dataset, essentially at the random-guessing baseline of 50.0. This is in stark contrast to human annotators who achieve 81.0 (cres) and 82.1 (cins) on the same dataset, indicating a large gap between the model's reward assignments and human preference judgments.