IC-479SAM's mask decoder exhibits attention drift to background or specific object parts under imprecise prompts, causing severe segmentation degradation

Qi Fan, Xin Tao, Lei Ke, Mingqiao Ye, Di ZHANG, Pengfei Wan, Yu-Wing Tai, Chi-Keung Tang

SourceStable Segment Anything Model

The paper systematically evaluates SAM (ViT-Large) across four segmentation datasets using prompts of varying quality: ground-truth boxes, noisy boxes (uniform noise scale 0.4), and point prompts with 1, 3, 5, or 10 points. With noisy box prompts, mIoU drops from 79.5 to 48.8 and the segmentation stability score (ST) is only 39.5; with a single point prompt, mIoU is 43.3 and ST is 45.1. Visualization of the token-to-image cross-attention in the second mask decoder layer reveals that imprecise prompts cause the decoder to activate image features biased toward the background or confined to specific object parts, a phenomenon the authors term attention drift. SAM 2 exhibits the same degradation pattern (mIoU 52.4 with noisy box, 46.7 with 1 point), confirming the failure mode is not specific to the original SAM checkpoint.

Evidence

correlational

Key metric

mIoU 79.5 (GT box) / 48.8 (noisy box) / 43.3 (1 point) / 78.7 (3 points) / 83.3 (5 points) / 84.8 (10 points); mBIoU 71.1 / 42.1 / 37.4 / 69.5 / 74.2 / 76.0; ST - / 39.5 / 45.1 / 79.3 / 84.7 / 87.5; SAM 2: mIoU 52.4 (noisy box), 46.7 (1 point), 81.1 (3 points)

Caveat

Analysis is on ViT-Large SAM specifically; attention drift visualization is from the second mask decoder layer only; 20 random prompts per image and prompt type were used for stability computation

Model

SAM, SAM 2

Concepts

Failure mode

Datasets

DIS [eval], ThinObject-5K [eval], COIFT [eval], HR-SOD [eval], MS COCO / COCO / COCO 2014 / COCO 2017 / COCO 20k / COCO-it / COCO-wl [eval], SGInW [eval], MESS [eval]

Extraction

automatic-extraction