Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Vanishing Gradients in Reinforcement Finetuning of Language Models
2024-01-16
· ICLR 2024 poster ·
anchor
Findings
IC-849
GPT-2 and T5-base exhibit vanishing expected gradients under RFT for inputs with small reward standard deviation, prevalent in 3 of 7 GRUE datasets, causing RFT to underperform SFT
IC-850
A partial SFT phase (40% of steps, 1% of samples) before RFT allows GPT-2 and T5-base to reach 96% of the reward achieved with full SFT+RFT, by reducing the number of inputs with vanishing gradients