SourceLocality Alignment Improves Vision-Language Models
The paper applies two interventions to the frozen ViT output before probing: (1) restricting the output head to only the [cls] token (or average embedding for models without one), and (2) anonymizing the ViT's output embeddings by removing learned positional embeddings and placing them in separate token positions. Both interventions degrade local probing performance, with the [cls]-only restriction causing the largest drop. This demonstrates that the patch embeddings in ViTs trained with image-level supervision (e.g., CLIP, IN1K) do carry useful local semantic information, rather than all information being collapsed into the [cls] token. However, the information is not perfectly organized for local prediction, which is why locality alignment still provides gains.