SourceVision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
The authors evaluate four CLIP model sizes (RN50, ViT-L/14, ViT-H/14, ViT-BigG/14) as reward models on the humanoid kneeling task. The EPIC distance to human labels improves smoothly and approximately log-linearly with model size. However, the human-evaluated success rate of the trained agent shows a sharp phase transition: RN50, ViT-L/14, and ViT-H/14 all achieve 0% success, while ViT-BigG/14 achieves 100%. Goal-baseline regularization improves the reward model across all sizes but is more impactful for smaller models. The authors note that predicting the exact point where the RL agent can learn the task is difficult, a common pattern in evaluating large foundation models.