SourceVision Transformers Need Registers
The LOSt algorithm, which detects objects by gathering information in attention maps, works well on DINO features (61.9 corloc on VOC 2007) but produces degraded performance on DINOv2 features. The paper reports DINOv2 achieving only 35.3/40.2/26.9 corloc on VOC 2007/2012 and COCO 20k, and DeiT-III even lower at 11.7/13.1/10.7. The authors attribute this to the high-norm outlier tokens corrupting the local feature maps that LOSt relies on. The qualitative analysis in Appendix C shows that the LOSt seed expansion score is disrupted by the artifacts in DINOv2 and DeiT-III, while for OpenCLIP the value projection appears to filter out the outliers, explaining its relatively better LOSt performance.