IC-668SAM's edge-oriented segmentation yields high recall but very low precision because it cannot distinguish object boundaries from interior edges

Cheng Shi, Sibei Yang

SourceThe Devil is in the Object Boundary: Towards Annotation-free Instance Segmentation using Foundation Models

The paper measures SAM-B and SAM-H on COCO val2017 for annotation-free instance segmentation. SAM achieves high average recall (AR100 of 36.9 for SAM-B, 44.5 for SAM-H) but very low average precision (AP of 1.4 and 0.9 respectively). The authors attribute this to SAM's edge-oriented approach: it segments all prominent edges including interior edges within objects (e.g., between an elephant's head and ear), and tends to assign higher confidence to smoother planes and smaller objects rather than to actual object instances. This makes it impossible to determine the confidence that a particular mask represents an object.

Evidence
observational
Key metric
SAM-B: AR100 36.9, AP 1.4; SAM-H: AR100 44.5, AP 0.9 (class-agnostic, COCO val2017, Table 1)
Model
SAM
Concepts
Failure mode
Datasets
MS COCO / COCO / COCO 2014 / COCO 2017 / COCO 20k / COCO-it / COCO-wl [eval]
Related findings
IC-667, IC-669
Extraction
automatic-extraction