CLIP / CLIP-ViT (LC)

OpenAI · 2021-02-26 · image, text · discriminative · anchor · artifact

Dual-encoder model that maps images and text into a shared space and scores their similarity.

Variants

CLIP RN50, CLIP RN101, CLIP RN50x4, CLIP ViT-B/32, CLIP ViT-B/16, CLIP ViT-L/14, CLIP ViT-L/14@336px, CLIP-B/32

Findings

• FX-001Pairwise Banzhaf interactions explain CLIP similarity more faithfully than single-score methods
• FX-002FIXLIP gives SigLIP-2 higher pointing-game recognition than CLIP at ViT-B/32 and ViT-B/16
• FX-003FIXLIP's strongest interaction in one CLIP example links doll to an image patch reading dollar
• IC-006The Retriever-Dictionary module improves object detection accuracy of YOLOv7, YOLOv9, Faster R-CNN, and Deformable DETR on COCO 2017 [context]
• IC-038Few embedding dimensions drive the modality gap in CLIP and SigLIP
• IC-039Object bias in CLIP and SigLIP is not correlated with performance on attribute tasks
• IC-040Information imbalance triggers both the modality gap and object bias in contrastive VLMs
• IC-041CLIP and SigLIP use the modality gap to control logit entropy
• IC-1011OpenAI CLIP loses approximately 8% zero-shot retrieval accuracy on 2021–2022 data compared to OpenCLIP models trained on data through 2022, while standard benchmarks show no such gap
• IC-1099AdamW-pretrained vision models (ViTs, ConvNeXt) have disproportionately large embedding-layer gradients at initialization, causing SGD fine-tuning to degrade OOD accuracy by up to 15% relative to AdamW
• IC-1105PAC-Bayes generalization bounds for discrete class prompts on CLIP are within a few percentage points of the actual test error across CIFAR-10, CIFAR-100, ImageNet, FMOW, and OfficeHome
• IC-1106CLIP prompts found by greedy search do not fit random labels: train and test error drop in tandem as the fraction of flipped labels increases, unlike a linear probe which achieves near-random accuracy
• IC-1201CLIP-ViT-L/14 image features support 200-way zero-shot EEG-based object recognition better than ViT-B/16 or ResNet-50 features when used as a frozen encoder in a contrastive learning framework
• IC-124Direct comparison of CLIP image embeddings with CLAP audio embeddings achieves near-chance retrieval, while logsumexp bridging through the shared language modality recovers 62% recall@10 on AudioSet
• IC-126CLIP and CLAP language representations are statistically indistinguishable from a uniform distribution on the hypersphere
• IC-1283CLIP's InfoNCE training objective is mathematically equivalent to performing generalized spectral clustering on the bipartite image-text pair graph
• IC-1290CLIP, DINO, and DINOv2 as zero-shot natural baselines score below the 50% chance level on Bongard-OpenWorld due to adversarial query selection
• IC-1389Video-language models do not significantly outperform image-language models on temporal reasoning tasks in VILMA
• IC-1390Proficiency tests reveal that a substantial portion of correct main-test predictions by VidLMs and ILMs are spurious rather than reflecting robust understanding
• IC-1469Progress on standard ImageNet generalization benchmarks is 2.5x faster than progress on crowdsourced global data (DollarStreet, GEODE) across 98 vision models
• IC-147CLIP's OOD performance on rendition domains is largely an artifact of domain contamination in its web-scale training data
• IC-1470Geographic disparities (Europe-Africa accuracy gap) are large across all 98 models and have more than tripled between least and best performing models on DollarStreet
• IC-1471Common robustness interventions (AugMix, CutMix, Deep AugMix, texture debiasing, antialiasing) and scaling of data or model size do not resolve geographic disparities in released vision models
• IC-1489State-of-the-art foundation models (CLIP, GPT-3.5-turbo, and others) score well below elementary students on multimodal K-12 STEM questions
• IC-1490Zero-shot CLIP is overconfident on STEM questions, with softmax confidence loosely related to actual accuracy
• IC-1491CLIP zero-shot performance on STEM saturates across model sizes, with only 3.6 points of variation from smallest to largest variant
• IC-150Second-order effects of CLIP's MLP neurons are concentrated in late layers (8–10 of 12 in ViT-B/32)
• IC-151Each CLIP neuron's second-order effect is approximately a single linear direction in the joint text-image space, significant for fewer than 2% of images
• IC-152CLIP's polysemantic neurons encode spurious correlations between unrelated concepts that can be exploited to generate adversarial misclassifications
• IC-154CLIP ViT-L/14 text embeddings fail to capture fine-grained visual class similarities, ranking rottweiler and doberman at position 828 behind unrelated pairs
• IC-1544The latent spaces of pretrained foundational models across vision and text are not related by a single class of geometric transformations; the optimal alignment depends on the specific model pair, architecture, and dataset.
• IC-1619DINOv2's patch-level features outperform CLIP and MAE for cross-image semantic feature matching
• IC-192CLIP's contrastive image-text training objective hinders its ability to rank or order images, yielding near-chance performance on ranking tasks in both zero-shot and fine-tuned settings
• IC-250Existing multimodal embedding models show highly uneven performance across MMEB's four meta-task categories, with VQA scores as low as 4.2 and overall scores ranging from 13.3 to 44.7
• IC-251CLIP's overall MMEB performance drops by 29.4% when task-specific instructions are prepended to queries, with classification degrading by 59.3%
• IC-280CLIP, OpenCLIP, and SigLIP exhibit intra-modal misalignment: intra-modal similarity comparisons are suboptimal for image-to-image and text-to-text retrieval
• IC-281SLIP's intra-modal self-supervised loss reduces intra-modal misalignment, making inter-modal inversion unnecessary for image retrieval
• IC-318CLIP backbones from different architectures (ViTs and ResNets) trained with the same data and objective exhibit complementary strengths, with an oracle per-image backbone selection improving zero-shot accuracy by up to 43.5% over the best single backbone
• IC-319Different CLIP backbones exhibit distinct robustness profiles to specific image perturbations, with each architecture being most resilient to a different transformation
• IC-331In Stable Diffusion v1.5, specific tokens in text prompts drive memorization, and GPT-4-based perturbation of high-attribution tokens reduces SSIM similarity to training images while maintaining CLIP score
• IC-357Off-the-shelf foundation models (DINO, CLIP, DINOv2, ViT) exhibit higher variance in their cosine similarity distributions than dataset-specific models, reducing the discriminative power of cosine similarity retrieval
• IC-415VLMs show a default shape bias (47.9-73.8%) that exceeds their vision encoders and vision-only models but falls short of human levels (96%), with the LLM component rather than the encoder responsible for suppressing one visual cue.
• IC-419CLIP ViT-B/16's layer-11 residual stream contains class-discriminative information in sparse SAE latent directions, and ablating class-specific top-k latents significantly degrades zero-shot classification accuracy
• IC-420CLIP ViT-B/16's SAE latent interpretability is depth-dependent: layer 11 encodes semantic object concepts while layers 2, 5, and 8 encode local shapes and attention patterns
• IC-447CLIP ViT-B/16 produces noisy saliency maps and contains only 42 concept detectors, indicating poor visual interpretability
• IC-448CLIP ViT-L/14 achieves 0% accuracy under 2/255 and 4/255 L-infinity adversarial perturbations across all 15 evaluation datasets
• IC-449CLIP ViT-B/16 Grad-CAM explanations are highly sensitive to input noise, with SSIM dropping from 91.18% to 70.58% as noise standard deviation increases from 1/255 to 9/255
• IC-469CLIP's global contrastive alignment causes attention on anatomically irrelevant regions in 3D CT, yielding limited zero-shot diagnostic accuracy (AUC 68.4 on 54 tasks)
• IC-470LOVT and MGCA, which use implicit cross-attention local alignment, show only marginal improvement over CLIP in 3D CT diagnosis (AUC 69.4 and 70.1 vs 68.4)
• IC-499ViT patch embeddings contain local semantic information beyond the [cls] token, as shown by performance degradation when restricting the output head to [cls] only or removing positional embeddings
• IC-548CLIP-B/32 exhibits progressively increasing layer-wise representation similarity in both its vision encoder and text encoder, and the pattern also holds across modalities
• IC-582Instruction-tuned MLLMs (InstructBLIP, mPLUG-Owl, Idefics) achieve significantly better brain alignment than vision-only ViT-H and perform comparably to or better than CLIP-text across whole visual cortex and five visual ROIs
• IC-604CLIP ViT-B/32's CIFAR-10 image embeddings approximately satisfy a multi-cluster structure with near-orthogonal class-mean features
• IC-642CLIP can infer contextual attributes (orientation, illumination, etc.) from images with approximately 74% accuracy on a binary task
• IC-643Conditioning CLIP on correct contextual attributes in the text prompt improves zero-shot classification accuracy across 13 image transformations
• IC-644CLIP relies on spurious features (background) as a shortcut in zero-shot classification, and conditioning on the correct background reduces this reliance
• IC-645CLIP, PickScore, and HPSv2 text embeddings share a common direction (cone effect) that captures text-irrelevant preferences, and the orthogonal component c⊥p better measures T2I alignment; CLIP's untrained common direction makes it ineffective for reward fine-tuning
• IC-666DINOv2, CLIP-vision, and VGG-19 representations all align with MEG brain responses, with DINOv2 showing particularly high retrieval performance for late brain activity after image offset
• IC-667CLIP's intermediate layer features encode object boundaries recoverable by k-means clustering, a property absent in shallow and deep layers
• IC-684CLIP ViT-B/32 misclassifies 99% of forest satellite images as ocean when the word 'ocean' is overlaid as text
• IC-685CLIP ViT-B/32 with a linear probe relies on gender as a spurious correlation for hair color, achieving only 15.85% accuracy on female gray hair
• IC-686Adversarial perturbations alter CLIP ViT-B/32's token representations most strongly starting around layer 10
• IC-731CLIP (ViT-B/32) achieves only 17.5 recall on video-text temporal alignment because it was trained on images and lacks video dynamics
• IC-737CLIP ViT-B/16 binarized dot products yield 0.50–0.58 accuracy on binary concept presence queries across five image classification datasets
• IC-739GPT-3.5-turbo-0613 combined with CLIP produces more faithful concept-salience pseudo-labels than LLaMA-2-13B-Chat, InstructBLIP, or LLaVA-1.5B on most of five datasets
• IC-763CLIP and OpenCLIP fall short of human discriminative accuracy on vision tasks, with performance dropping substantially under hard negatives
• IC-784Pre-trained scoring models (CLIP Score, HPS, Image Reward, Pick Score) underperform on domain-specific fine-tuned diffusion models
• IC-786CLIP-ViT (LC) achieves 0.87 accuracy and 0.91 average precision on fake image detection
• IC-788CLIP ViT-B/32 fails to retrieve the correct image even when the generated target caption is well-aligned with the ground-truth image
• IC-789CLIP retrieval quality in zero-shot compositional image retrieval scales log-linearly with model size from approximately 150M to 2.5B parameters
• IC-834CLIP zero-shot predictions exhibit high equal opportunity difference when target and sensitive attributes are intrinsically dependent
• IC-835CLIP zero-shot predictions exhibit large worst-group accuracy gaps due to spurious correlations on Waterbirds and CelebA
• IC-836CLIP zero-shot predictions exhibit demographic bias on FairFace when using attribute-unrelated text prompts
• IC-837CLIP ViT-L/14 embeds more target-attribute information and less sensitive-attribute information than CLIP ResNet-50 on Waterbirds
• IC-847CLIP ViT-B/32 CLIPScore achieves only ρ=0.276 / τ=0.191 correlation with human 1-5 likert T2I alignment ratings on TIFA160
• IC-939CLIP reward landscapes are well-shaped for photorealistic environments but poorly shaped for abstract renderings
• IC-940CLIP can specify 5 of 8 complex humanoid tasks from single-sentence prompts, failing on tasks requiring discrimination of subtle body-pose differences
• IC-941CLIP reward model quality scales with model size, with a sharp phase transition between ViT-H/14 and ViT-BigG/14 for the humanoid kneeling task
• IC-983In Stable Diffusion's CLIP text-encoder, knowledge about all visual attributes is localized to a single causal state: the first self-attention layer at the last subject token
• IC-984CLIP ViT-B/16's representation space does not reliably preserve semantic similarity as measured by shared image tags

Shared mechanisms

• Depth-dependent structurealso in Baichuan 2, BERT, BLIP-2, BLOOM, Chameleon, DeepFloyd IF, DeiT-III, DINO, DINOv2, Falcon, Gemma, Gemma 2, GPT-2, GPT-J, GPT-NeoX-20B, Griffin, I3D, Idefics, InstructBLIP, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Phi, MAE, MAE-B/16, Mamba, MiniGPT-4, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, mPLUG-Owl, MPT, MultiBERTs, MViT V2, OLMo / OLMo base, OpenCLIP, OPT, Phi-2, Pythia, Qwen 2, Qwen2-VL, Qwen2.5, RoBERTa / RoBERTa-L, RWKV, SALMONN, SAM, SlowFast, Stable Diffusion, Swin Transformer, TerraMind, TimesFormer, TSM, Uniformer, Vicuna, VideoMAE, ViT, X3D, Yi
• Distance preservationalso in CoPlace, DINO, DINOv2, Gemma, Gemma 2, ImageBind, LanguageBind, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, MAE, MAE-B/16, OpenCLIP, Pythia, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SigLIP, SLIP, TerraMind, ViT
• Explanation faithfulnessalso in BakLLaVA, CF2, Claude 3, Claude 3.5, DRUM, Fuyu, GEM, Gemini 1.5 / Gemini Pro 1.5, Gemma 2, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, Idefics, Llama 3.1, LLaVA-NeXT / LLaVA 1.6, MAE-B/16, MobileNetV2, mPLUG-Owl3, OpenFlamingo, PGExplainer, Pythia, RCExplainer, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SigLIP, SigLIP-2, Stable Diffusion, TAGExplainer, ViT
• Failure modealso in AASIST, ADM, Aegis-Guard-Defensive, Alpaca, AnyLoc, AutoTikZ / DataTikZ, Baichuan, Baichuan 2, Baichuan2-13B, BakLLaVA, Bard, BEiT, BERT, BingChat, BLIP, BLIP-2, BLOOM, BSRNet, CF2, Chat-UniVi-7B, ChatGLM-6B / ChatGLM-6b-2, ChatGLM2, ChatGPT, CLAP, Claude 1.3, Claude 2.0, Claude 2.1, Claude 3, Claude 3.5, CLEAR, CLIP4Clip, CLIPBERT, CLIPCap, CLMBR-T-BASE, CloFNet, Code Llama, CodeGeex2, CodeGen, CodeLlama-13B, CodeLlama-34B, CogVLM2, Cohere Command R, CoMEt, Command R+, CONCH, CycleGAN, DALL-E, DALL·E 2, DALL·E 3, DASR, DECAF, DeepSeek-2-Chat, DeepSeek-2-Coder, DeepSeek-V2-0628, DeepSeek-VL, DeepSeek-VL2, DeiT, DeiT-III, Depth Anything, DETR-R50, DimeNet++, DINO, DINO-FocalNet-Large, DINOv2, EGNN, Emu2, EquiformerV2, ESCN, ESM-2, ESM3, ESRGAN, EVA-CLIP, EVE, Falcon, Faster R-CNN / Faster R-CNN R50 / Faster R-CNN X101, FLAN-T5, Florence-2, FLUX / FLUX1, FoveaBox, Fuyu, Galactica-6.7B, GAT, 3D Gaussian Splatting, GCN, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GIN, GLIDE, GLM-4, GLM-4V, GloVe, GP-UNIT, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4.1, GPT-4o, GPT-J, GPT-NeoX-20B, GraphSAGE, Grounding DINO, Guanaco, GVP, Hawkeye, HiFaceGAN, HPS, HuggingGPT, IDDPM, Idefics, Idefics2, ImageBind, ImageBind-LLM-7B, Imagen Video, ImageReward, 12-in-1, InstructBLIP, InstructGPT, InternLM-2.5-7B, InternLM-XComposer2-VL, InternVideo, InternVL-1.5, InternVL2, Koala, LegalBERT, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, Llama-3-2-Vision, LLaMA-Adapter v2, Llama Guard, Llama Guard 2, Llama-Guard 3, Llama-VID, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, LLaVA-OneVision, LongVA-7B, LOVT, LWM-1M-JAX, MACE, MAE, Med-Flamingo, Merlot Reserve, MGCA, MiDaS, MiniCPM-V, MiniGPT-4, Mip-Splatting, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mistral Large 2, Mistral Large V2, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, MobileNetV2, Molmo, MolmoE-7B, Momentor, Moondream2, Moonshot-v1-8k, mPLUG-2, mPLUG-Owl, mPLUG-Owl3, mPLUG-Owl2, MPT, MSA Transformer, MultiBERTs, Nova Canvas, Nova Lite, Nova Pro, O1 / OpenAI-o1-preview, O3, O4-mini, OLMo / OLMo base, OneLLM, OpenAI Moderation, OpenChat-3.5-0106, OpenCLIP, OpenFlamingo, OPT, Otter, Otter-7B, PaLM 2, PaLM 62B, PandaGPT-7B, PerSAM, Phi-3, Phi-3.5 Mini Instruct, PickScore, PLIP, Prismatic, ProGen-2, Pythia, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen-VL, Qwen2.5, Qwen2-Audio, R2D2, RadFM, RCExplainer, RD-SR, RealESRNet, Reprover, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, RetinaNet-R50, RivaGAN, RS-LDS, RT-1, RT-2, SALMONN, SAM, SAM 2, SAULLM 54B, Scaffold-GS, SchNet, Seed-LLaMA-8B, SGC, SIREN, Sketch Transformer, SLD-max, SLD-medium, SLD-strong, SLDS, SLIMG, SpeechGPT, SphereNet, SRResNet, Stable Diffusion, StackLLaMA, Starcoder, StegaStamp, StyleGAN2-ADA, Swin Transformer, T5, TD-MPC, TerraMind, TimeChat, TranceptionEVE, TreeRing, Tulu 2, UnifiedQA, UniPerceiver, UNITER, UniVL, Van Gogh Diffusion, VERA, VGG / VGG13, Vicuna, Video-Chat-7B, Video-ChatGPT, Video-LLaMA, Video-LLaMA-2-13B, Video-LLaVA, VideoCLIP, ViLA-8B, ViLBERT, VindLU, VioLET, ViRTex, ViT, ViV1T, VTG-LLM, WildGuard, Wizardlm, X-CLIP, X-InstructBLIP-7B, XGen-MM, Xlm-R, Zephyr-7B-beta
• Feature interactionalso in Aquila-7B, GPT-2, LLaMA, OPT, Sybil
• Linear representationalso in BLOOM, Cambrian-1, Chameleon, DINOv2, EVA-CLIP, Falcon, Gemma, Gemma 2, GPT-2, GPT-J, HPSv2, ImageBind, InstructBLIP, LanguageBind, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, MAE, Mamba, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, OLMo / OLMo base, OpenCLIP, Phi-3, PickScore, Pythia, Qwen2-VL, Qwen2.5, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SALMONN, SAM, SigLIP, TerraMind, Tulu 2, Vicuna, ViT
• Method artefactalso in Baichuan, ConvNeXt, EfficientNet, Falcon, Gemma, Gemma 2, GPT-2, GPT-3.5 / ChatGPT-3.5, GPT-4o, GPT-J, InternLM, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, MAP-NEO, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, OLMo / OLMo base, OpenCLIP, OpenLLaMA, OPT, Pythia, Qwen1.5, Qwen 2, Qwen2.5, RedPajama, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, Skywork, Stable Diffusion, StableLM, TerraMind, ViT, Yi, ZiYA2
• Scale-dependent behaviouralso in Aquila-7B, BEiT, BERT, BLOOM, Claude 2.1, Claude 3, Claude 3.5, Code Llama, CodeGen, Cohere Command R, DeepSeek LLM, DeepSeekMoE, DeiT-III, DINO, DINOv2, EquiformerV2, ESCN, Falcon, FLAN-T5, Gemini 1.0 Pro, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, GPT-Neo, I3D, Idefics, InternLM-2.5-7B, InternLM-XComposer2-VL, InternLM2, InternVL-1.5, InternVL2, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, LongVA-7B, MAE, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Moirai, MPT, MViT V2, O1 / OpenAI-o1-preview, OLMo / OLMo base, OpenCLIP, OpenFlamingo, OpenLLaMA, OPT, PaLM 2, Phi-3, Platypus2-Instruct-70B, Pythia, Qwen, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen2.5, RedPajama-INCITE, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SlowFast, Solar 10.7B, Stable Diffusion, StableLM, Swin Transformer, T5, TerraMind, text-ada-001, TigerBot-70B-Chat, TimesFormer, TSM, Tulu 2, Uniformer, Vicuna, VideoMAE, ViLA-8B, Wizardlm, X3D, XGLM, Xwin-LM-70B-v0.1, Yi
• Shortcutalso in BakLLaVA, BLIP-2, Claude 3, Claude 3.5, CLIP4Clip, CLIPBERT, DALL·E 2, DALL·E 3, DeepSeek-VL2, Eurus-RM-7B, Falcon, FLUX / FLUX1, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, iFlytekSpark-13B, InstructBLIP, Internlm2-Reward, InternVideo, InternVL2, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama-3-2-Vision, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, Med-Flamingo, Merlot Reserve, MiniGPT-4, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Molmo, mPLUG-2, mPLUG-Owl3, Nova Canvas, O1 / OpenAI-o1-preview, OpenCLIP, OPT, Otter, PaLM 62B, Pythia, Qwen, Qwen 2, Qwen-VL, Qwen2.5, RadFM, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, Stable Diffusion, Swin Transformer, Sybil, TerraMind, Tulu 2, UniPerceiver, UniVL, Vicuna, Video-LLaMA, VideoCLIP, VindLU, VioLET, X-CLIP