SourceThe "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities
The paper evaluates whether the image encoder of CLIP and the audio encoder of CLAP can be directly compared via normalized dot product for audio-visual retrieval. Direct comparison yields only 14% recall@10 on AudioSet, barely above the 10% chance level for a 10-way task. Applying the logsumexp (monte carlo) method, which marginalizes over the intermediate language modality using the AudioSet ontology as the sample space, raises performance to 62% recall@10. This demonstrates that two independently trained contrastive models cannot be naively plugged together without an explicit bridging procedure.