SourceFacing the Elephant in the Room: Visual Prompt Tuning or Full finetuning?
The paper compares visual prompt tuning (VPT) against full fine-tuning (FT) on the released ViT-B/16 checkpoint pretrained on ImageNet-21k, across all 19 tasks in the VTAB-1k benchmark. VPT wins 16/19 tasks (6/7 natural, 2/4 specialized, 8/8 structured). The advantage is strongest when the downstream task objective differs substantially from classification (e.g., counting, distance) or when the data distribution is similar to ImageNet. As the downstream training set grows from 400 to 20,000 samples, the gap closes and FT surpasses VPT in 9 of 12 plotted cases. One-shot experiments confirm VPT's advantage in low-data regimes (13/19 tasks). Similar patterns are observed for Swin-Base, MAE-pretrained ViT-B/16, and MoCo v3-pretrained ViT-B/16 in the appendix.