SourceProtein Language Model Fitness is a Matter of Preference
Across 217 deep mutational scan studies from ProteinGym, the pseudo log likelihood (PLL) of the wild type sequence predicts the Spearman correlation of zero-shot mutation effect predictions for both ESM-2 and ProGen-2 at multiple scales. Performance improves as likelihood increases from very low values, but degrades again at very high likelihoods, producing a concave-down parabola. This effect is most pronounced in larger models (ESM-2 15B, ProGen-2 XL), where sequences with probabilities near 1 show the greatest degradation. Additionally, the explanatory power of likelihood for performance decreases as model size increases, an inverse scaling law.