LanguageBind

text, video · discriminative · anchor

Findings

• IC-124Direct comparison of CLIP image embeddings with CLAP audio embeddings achieves near-chance retrieval, while logsumexp bridging through the shared language modality recovers 62% recall@10 on AudioSet [compared-to]
• IC-125LanguageBind's direct evaluation achieves 70% recall@10 on AudioSet, empirically validating that the inner product between unpaired modality representations recovers the correct probability ratio
• IC-390The parallelotope volume of modality embeddings from LanguageBind, VAST, and Valor on MSR-VTT is strongly correlated with their downstream R@1 retrieval performance

Shared mechanisms

• Distance preservationalso in CLIP / CLIP-ViT (LC), CoPlace, DINO, DINOv2, Gemma, Gemma 2, ImageBind, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, MAE, MAE-B/16, OpenCLIP, Pythia, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SigLIP, SLIP, TerraMind, ViT
• Linear representationalso in BLOOM, Cambrian-1, Chameleon, CLIP / CLIP-ViT (LC), DINOv2, EVA-CLIP, Falcon, Gemma, Gemma 2, GPT-2, GPT-J, HPSv2, ImageBind, InstructBLIP, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, MAE, Mamba, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, OLMo / OLMo base, OpenCLIP, Phi-3, PickScore, Pythia, Qwen2-VL, Qwen2.5, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SALMONN, SAM, SigLIP, TerraMind, Tulu 2, Vicuna, ViT