IC-041CLIP and SigLIP use the modality gap to control logit entropy

Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, Thomas Brox

SourceTwo Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models

The paper finds that the modality gap can serve as a mechanism to adjust the entropy (uncertainty) of the output logits. When the temperature parameter is frozen, models increase the modality gap to achieve the same entropy as models with a learnable temperature. This suggests the gap is a flexible tool for controlling the logit entropy on a per-sample basis, rather than a simple bug.

Evidence

interventional

Key metric

Modality gap (l2m & rmg) and entropy over logits are similar for frozen temperature and learnable temperature settings after fine-tuning.

Caveat

The paper does not claim a causal relationship; it states that the changes to the embedding are 'measurable by the modality gap'.

Model

CLIP / CLIP-ViT (LC)

Datasets

CC12M [train]

Extraction

automatic-extraction