IC-120YOLO-World can replace SAM as a 2D crop generator in OpenMask3D's pipeline with nearly equivalent mAP but 1.76x faster inference

Mohamed El Amine Boudjoghra, Angela Dai, Jean Lahoud, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan

SourceOpen-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation

The paper tests four 2D detectors (SAM, YOLOv8, RT-DETR, YOLO-World) as alternatives to SAM for generating bounding-box crops in OpenMask3D's CLIP feature aggregation pipeline, using ground-truth 3D mask proposals on the Replica dataset. YOLO-World achieves mAP 32.5 compared to SAM's 33.0, a gap of only 0.5 points, while reducing per-scene inference time from 675.6 s to 384.29 s. YOLOv8 and RT-DETR show larger accuracy drops (21.1 and 28.4 respectively), indicating that YOLO-World's open-vocabulary zero-shot detection quality is the key factor enabling it to match SAM.

Evidence

correlational

Key metric

mAP 32.5 (YOLO-World) vs 33.0 (SAM), time/scene 384.29 s vs 675.6 s on Replica with ground-truth 3D masks

Caveat

The comparison uses ground-truth 3D mask proposals, so the finding is about crop generation quality in isolation, not about the full end-to-end pipeline. The 0.5 mAP gap, while small, is not zero.

Model

YOLO-World, SAM, YOLOv8, RT-DETR

Datasets

Replica [eval]

Methods

OpenMask3D [primary], CLIP [supporting]

Related work

OpenMask3D [builds-on]

Extraction

automatic-extraction