SourceFrozen Transformers in Language Models Are Effective Visual Encoder Layers
The paper tests the last transformer block from OPT at five scales (125M, 350M, 1.3B, 2.7B, 6.7B) appended to ViT-s, trained for 100 epochs following DeiT. The two smallest OPT blocks (125M, 350M) actually reduce accuracy below the ViT-s baseline, while the three larger blocks (1.3B, 2.7B, 6.7B) all improve it. The authors note that the small OPT blocks even produce NaN loss values at the original DeiT learning rate, requiring a 5x reduction for stable training, further indicating that sufficient scale is a prerequisite for the phenomenon.