IC-1496A single frozen transformer block from LLaMA-7B consistently improves performance across diverse visual tasks when appended to existing visual encoders
The paper appends one frozen transformer block from LLaMA-7B (the last block by default) to various visual encoders and trains the rest of the network. Across image classification (ViT), point cloud classification (Point-BERT), action recognition (ViT on SSV2), motion forecasting (VectorNet, MMTransformer), and vision-language tasks (METER, SQA3D), the LLaMA block yields consistent gains. Ablations with a matched-capacity MLP and with randomly initialized LLaMA blocks confirm the improvement comes from the pre-trained LLaMA weights rather than added parameters. Fine-tuning the LLaMA block instead of freezing it degrades performance due to overfitting.
Experiments are designed for diversity of tasks under fair comparisons rather than state-of-the-art performance; computational resources limited batch sizes (e.g., 24 clips vs 384 in VideoMAE), so absolute numbers are lower than original papers.