IC-1226SAM's ViT-B encoder achieves 54.2% ImageNet-1k linear probing accuracy versus 67.7% for MAE's ViT-B, indicating its segmentation pretraining impairs high-level semantic representation

Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, Chun Yuan

SourceConvolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model

The authors freeze SAM's ViT-B image encoder and train only a linear classification head on ImageNet-1k. Under the same protocol, a ViT-B pretrained with MAE achieves 67.7% accuracy. SAM's encoder, despite being initialized from an MAE encoder, scores only 54.2%. The authors attribute this 13.5-point gap to SAM's pretraining objective, which is binary foreground-background mask prediction without explicit semantic class labels, causing the encoder to optimize for within-image objectness rather than cross-image semantic consistency.

Evidence

correlational

Key metric

54.2% vs. 67.7% ImageNet-1k linear probing accuracy (SAM ViT-B vs. MAE ViT-B)

Model

SAM, MAE

Concepts

Linear representation

Datasets

ImageNet-1k / ImageNet / ImageNet-1k-val / ImageNet-Val [eval]

Methods

Linear Probing / Ridge regression linear probing / Linear probe / Linear probe fine-tuning / Linear regression probing / Linear ridge regression probes / Supervised probing / ERM linear probe [primary]

Related findings

IC-1227

Extraction

automatic-extraction