IC-146Pythia models show increasing robustness to off-policy RLHF data as policy size scales from 410M to 2.8B

Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, Aaron Courville

SourceAsynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models

The paper tests how Pythia models of three sizes (410M, 1B, 2.8B) perform under varying levels of off-policy data during RLHF training on TLDR summarization. By generating n mini-batches before each update (n=1 to 64), the authors create a spectrum from fully on-policy to highly off-policy training. They find that larger models maintain performance better under off-policy conditions: the 410M model has two points (n=16, 32) far from the optimal win-rate/KL tradeoff with wide spread, whereas the 2.8B model's worst point (n=64) remains close to optimal. This indicates that model capacity is a key factor in tolerating stale training data during RLHF.

Evidence

correlational

Key metric

410m has two points (n = 16, 32) far from the optimal area and a wide spread, whereas 2.8b's worst point (n = 64) is still quite close to optimal

Caveat

Models are SFT-fine-tuned before RLHF, so the finding is about the Pythia family's property as initialized by standard SFT on TLDR. The specific win-rate and KL values for each n are shown only in Figure 5 (left) and not printed as numbers in the text.

Model

Pythia

Concepts

Scale-dependent behaviour

Datasets

TL;DR [eval]

Methods

Online DPO [primary], RLOO [compared-to]

Extraction

automatic-extraction