SourceVision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
The paper evaluates CLIP's cosine-similarity reward in CartPole and MountainCar environments. In CartPole, the reward is well-shaped around the goal regardless of rendering. In the default (abstract) MountainCar, the reward landscape is poorly shaped and the agent cannot be trained to solve the task. When the MountainCar environment is re-rendered with realistic textures, the reward becomes well-shaped and the agent achieves 100% success. The authors attribute this to CLIP's visual encoder being trained on photorealistic images, so abstract renderings fall outside its effective input distribution.