IC-849GPT-2 and T5-base exhibit vanishing expected gradients under RFT for inputs with small reward standard deviation, prevalent in 3 of 7 GRUE datasets, causing RFT to underperform SFT
The paper proves that the expected gradient for an input under RFT vanishes when its reward standard deviation is small (Theorem 1). Empirically, on the GRUE benchmark, three of seven datasets (NarrativeQA, ToTTo, CommonGen) contain a substantial fraction of inputs with near-zero pretrain reward standard deviation under GPT-2 or T5-base, while their expected reward is suboptimal. RFT (PPO) has limited impact on the rewards of these inputs, as quantified by a high Pearson correlation between pretrain reward std and absolute reward change (0.48 for RFT vs 0.05 for SFT on NarrativeQA). In datasets where small-std inputs are more prevalent, RFT achieves worse reward relative to SFT. Conventional heuristics (increased learning rate, temperature, entropy regularization) do not alleviate the problem.
The authors note that a causal relation between small reward std prevalence and RFT underperformance cannot be inferred from the GRUE experiments alone due to possible confounding factors such as insufficient exploration; the controlled experiments and theory are needed to establish causality. Experiments use small-to-moderate models and do not incorporate reward functions learned from human feedback.