IC-1156The VPT advantage over FT for ViT-B/16 is not explained by overfitting resistance or additional optimization dimensions; the specific feature-preservation mechanism of VPT is the key factor

Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, Dongfang Liu

SourceFacing the Elephant in the Room: Visual Prompt Tuning or Full finetuning?

The paper tests two hypotheses for why VPT outperforms FT on ViT-B/16. First, overfitting: training/testing loss curves show that 8/9 structured tasks overfit for both FT and VPT, while only 1/10 natural+specialized tasks overfit, so overfitting cannot explain VPT's advantage in the latter cases. Second, additional dimensions helping escape local minima: the 'mixed' method (VPT + full FT) and 'ft-then-pt' method both add dimensions but underperform VPT (mixed: 72.80/79.78/42.94; ft-then-pt: 78.20/82.70/50.38 vs VPT: 79.53/82.65/56.09 on natural/specialized/structured). Grad-CAM and Integrated Gradients visualizations show VPT focuses on task-relevant regions more effectively than FT. The conclusion is that VPT's unique combination of preserving the frozen backbone features while adding a small set of learnable prompts is the pivotal mechanism.

Evidence

correlational

Key metric

Overfitting in 8/9 structured tasks for both methods; only 1/10 natural+specialized tasks overfit; mixed mean accuracy 72.80/79.78/42.94 vs VPT 79.53/82.65/56.09 (natural/specialized/structured); ft-then-pt 78.20/82.70/50.38

Caveat

The authors note that the visualization-based evidence (Grad-CAM, IG) is qualitative and that the connection between attention visual evidence and performance is not fully established; they cite TOAST (Wang et al., 2023) as relevant work on this gap.

Model

ViT ViT-B/16

Datasets

VTAB-1k [eval]

Methods

Visual Prompt Tuning [primary], Grad-CAM [supporting], Integrated Gradients / Integral of gradients [supporting]

Related work

TOAST [context]

Related findings

IC-1155

Extraction

automatic-extraction