SourceConvolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model
The authors freeze SAM's ViT-B image encoder and train only a linear classification head on ImageNet-1k. Under the same protocol, a ViT-B pretrained with MAE achieves 67.7% accuracy. SAM's encoder, despite being initialized from an MAE encoder, scores only 54.2%. The authors attribute this 13.5-point gap to SAM's pretraining objective, which is binary foreground-background mask prediction without explicit semantic class labels, causing the encoder to optimize for within-image objectness rather than cross-image semantic consistency.