IC-1498The benefit of frozen LLM transformer blocks for visual encoding is scale-dependent: OPT blocks below 1.3B parameters degrade ViT-s performance while blocks at 1.3B and above improve it
The paper tests the last transformer block from OPT at five scales (125M, 350M, 1.3B, 2.7B, 6.7B) appended to ViT-s, trained for 100 epochs following DeiT. The two smallest OPT blocks (125M, 350M) actually reduce accuracy below the ViT-s baseline, while the three larger blocks (1.3B, 2.7B, 6.7B) all improve it. The authors note that the small OPT blocks even produce NaN loss values at the original DeiT learning rate, requiring a 5x reduction for stable training, further indicating that sufficient scale is a prerequisite for the phenomenon.
Experiments use 100-epoch training (ablation setting) rather than the full 300-epoch protocol; small OPT models required a 1/5 learning rate reduction to avoid NaN loss.