Light Dark text, image · generative · anchor
Open-weight vision-language model pairing a CLIP vision encoder with an open language model through a projection layer, cited in the corpus at 7B and 13B.
Note anchor found by search rather than in a citing paper, and checked against this entry's own description before it was recorded: "Improved Baselines with Visual Instruction Tuning" Variants LLaVA-1.5 7B , LLaVA-1.5 13B , LLaVA-v1.5-7B , LLaVA-v1.5-13B Findings IC-018 Object information is localized to specific visual tokens in LLaVA-1.5 IC-019 Visual token representations in LLaVA-1.5 evolve to align with interpretable text tokens IC-020 LLaVA-1.5 extracts object information directly from visual tokens to the last token in mid-late layers IC-025 LMMs exhibit poor fine-grained perception in locating individual characters on original oracle bones IC-027 LMM performance in deciphering oracle bone inscriptions is comparable to untrained humans for common characters but declines for rarer and structurally complex characters IC-042 No single knowledge editing method excels across all criteria when editing visual and user-specific knowledge in LMMs. IC-106 Logit lens on LLaVA and InstructBLIP image representations shows higher internal confidence for objects present in the image than for hallucinated objects IC-107 Linear orthogonalization of LLaVA and InstructBLIP image features against text embeddings removes hallucinated objects at 83-86% individual rate versus 7-16% for correctly detected objects IC-108 Per-patch logit lens confidence in LLaVA localizes objects spatially, achieving mAP 79.90 on ImageNet segmentation, 8.03% above raw VLM attention IC-116 In LLaVA-7B, multi-head attention modules drive hallucination more than MLP modules, and targeted intervention on specific hallucination heads reduces the hallucination rate by up to 1.7x IC-117 Hallucination heads in LLaVA-7B and MiniGPT-4 are concentrated in the middle and deeper layers of the transformer IC-118 Hallucination heads in LLaVA-7B and MiniGPT-4 allocate 4.75x more attention to text tokens than image tokens, and this pattern is inherited from the base language model IC-119 The number of salient hallucination heads decreases as model size increases within the LLaVA family IC-128 Released VLMs exhibit sycophancy, agreeing with incorrect user opinions while ignoring visual evidence, with LLaVA-1.5 showing the highest rate (94.6%) and InternLM-XComposer2-VL-1.8B the lowest (28.8%) IC-130 Amplifying visual-token attention in high layers (16-32) of released VLMs reduces sycophancy while preserving VQA accuracy, indicating that insufficient high-layer visual attention is a key cause of sycophancy IC-163 LLaVA-1.5, LLaVA-Next, and GPT-4V show near-zero accuracy on GUI grounding benchmarks while achieving 50-85 on general image grounding (RefCOCO+), indicating a failure mode specific to GUI grounding scenarios IC-276 All 14 evaluated VLMs show a large gap between average-case and worst-case accuracy on DynaMath variants, with worst-case at or below 50% of average-case, and the failures are systematic rather than random IC-414 LLaVA-1.5-7B and LLaVA-1.5-13B exhibit severe performance degradation when H2O KV cache compression is applied in multimodal settings IC-900 BLIP-2, MiniGPT-4, and LLaVA-1.5 show degraded zero-shot VQA accuracy on underspecified questions, with absolute improvements of 1.14–7.94% when questions are augmented with visually-grounded details Shared mechanisms Depth-dependent structure also in Baichuan 2 , BERT , BLIP-2 , BLOOM , Chameleon , CLIP / CLIP-ViT (LC) , DeepFloyd IF , DeiT-III , DINO , DINOv2 , Falcon , Gemma , Gemma 2 , GPT-2 , GPT-J , GPT-NeoX-20B , Griffin , I3D , Idefics , InstructBLIP , LLaMA , Llama 2 / Llama 2 base , Llama 3 , Llama 3.1 , Llama 3.2 , Llama-3.2-3B , LLaVA , LLaVA-Phi , MAE , MAE-B/16 , Mamba , MiniGPT-4 , Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1 , Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b , mPLUG-Owl , MPT , MultiBERTs , MViT V2 , OLMo / OLMo base , OpenCLIP , OPT , Phi-2 , Pythia , Qwen 2 , Qwen2-VL , Qwen2.5 , RoBERTa / RoBERTa-L , RWKV , SALMONN , SAM , SlowFast , Stable Diffusion , Swin Transformer , TerraMind , TimesFormer , TSM , Uniformer , Vicuna , VideoMAE , ViT , X3D , Yi Distance preservation also in CLIP / CLIP-ViT (LC) , CoPlace , DINO , DINOv2 , Gemma , Gemma 2 , ImageBind , LanguageBind , LLaMA , Llama 2 / Llama 2 base , Llama 3 , Llama 3.1 , Llama-3.2-3B , LLaVA-Med , MAE , MAE-B/16 , OpenCLIP , Pythia , ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN , SigLIP , SLIP , TerraMind , ViT Failure mode also in AASIST , ADM , Aegis-Guard-Defensive , Alpaca , AnyLoc , AutoTikZ / DataTikZ , Baichuan , Baichuan 2 , Baichuan2-13B , BakLLaVA , Bard , BEiT , BERT , BingChat , BLIP , BLIP-2 , BLOOM , BSRNet , CF2 , Chat-UniVi-7B , ChatGLM-6B / ChatGLM-6b-2 , ChatGLM2 , ChatGPT , CLAP , Claude 1.3 , Claude 2.0 , Claude 2.1 , Claude 3 , Claude 3.5 , CLEAR , CLIP / CLIP-ViT (LC) , CLIP4Clip , CLIPBERT , CLIPCap , CLMBR-T-BASE , CloFNet , Code Llama , CodeGeex2 , CodeGen , CodeLlama-13B , CodeLlama-34B , CogVLM2 , Cohere Command R , CoMEt , Command R+ , CONCH , CycleGAN , DALL-E , DALL·E 2 , DALL·E 3 , DASR , DECAF , DeepSeek-2-Chat , DeepSeek-2-Coder , DeepSeek-V2-0628 , DeepSeek-VL , DeepSeek-VL2 , DeiT , DeiT-III , Depth Anything , DETR-R50 , DimeNet++ , DINO , DINO-FocalNet-Large , DINOv2 , EGNN , Emu2 , EquiformerV2 , ESCN , ESM-2 , ESM3 , ESRGAN , EVA-CLIP , EVE , Falcon , Faster R-CNN / Faster R-CNN R50 / Faster R-CNN X101 , FLAN-T5 , Florence-2 , FLUX / FLUX1 , FoveaBox , Fuyu , Galactica-6.7B , GAT , 3D Gaussian Splatting , GCN , Gemini , Gemini 1.5 / Gemini Pro 1.5 , Gemma , Gemma 2 , GIN , GLIDE , GLM-4 , GLM-4V , GloVe , GP-UNIT , GPT-2 , GPT-3 / GPT base , GPT-3.5 / ChatGPT-3.5 , GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report , GPT-4.1 , GPT-4o , GPT-J , GPT-NeoX-20B , GraphSAGE , Grounding DINO , Guanaco , GVP , Hawkeye , HiFaceGAN , HPS , HuggingGPT , IDDPM , Idefics , Idefics2 , ImageBind , ImageBind-LLM-7B , Imagen Video , ImageReward , 12-in-1 , InstructBLIP , InstructGPT , InternLM-2.5-7B , InternLM-XComposer2-VL , InternVideo , InternVL-1.5 , InternVL2 , Koala , LegalBERT , LLaMA , Llama 2 / Llama 2 base , Llama 3 , Llama 3.1 , Llama 3.2 , Llama-3.2-3B , Llama-3-2-Vision , LLaMA-Adapter v2 , Llama Guard , Llama Guard 2 , Llama-Guard 3 , Llama-VID , LLaVA , LLaVA-Med , LLaVA-NeXT / LLaVA 1.6 , LLaVA-OneVision , LongVA-7B , LOVT , LWM-1M-JAX , MACE , MAE , Med-Flamingo , Merlot Reserve , MGCA , MiDaS , MiniCPM-V , MiniGPT-4 , Mip-Splatting , Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1 , Mistral Large 2 , Mistral Large V2 , Mixtral , Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b , MobileNetV2 , Molmo , MolmoE-7B , Momentor , Moondream2 , Moonshot-v1-8k , mPLUG-2 , mPLUG-Owl , mPLUG-Owl3 , mPLUG-Owl2 , MPT , MSA Transformer , MultiBERTs , Nova Canvas , Nova Lite , Nova Pro , O1 / OpenAI-o1-preview , O3 , O4-mini , OLMo / OLMo base , OneLLM , OpenAI Moderation , OpenChat-3.5-0106 , OpenCLIP , OpenFlamingo , OPT , Otter , Otter-7B , PaLM 2 , PaLM 62B , PandaGPT-7B , PerSAM , Phi-3 , Phi-3.5 Mini Instruct , PickScore , PLIP , Prismatic , ProGen-2 , Pythia , Qwen1.5 , Qwen 2 , Qwen 2.5 72B Instruct , Qwen2-VL , Qwen-Audio , Qwen-VL , Qwen2.5 , Qwen2-Audio , R2D2 , RadFM , RCExplainer , RD-SR , RealESRNet , Reprover , ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN , RetinaNet-R50 , RivaGAN , RS-LDS , RT-1 , RT-2 , SALMONN , SAM , SAM 2 , SAULLM 54B , Scaffold-GS , SchNet , Seed-LLaMA-8B , SGC , SIREN , Sketch Transformer , SLD-max , SLD-medium , SLD-strong , SLDS , SLIMG , SpeechGPT , SphereNet , SRResNet , Stable Diffusion , StackLLaMA , Starcoder , StegaStamp , StyleGAN2-ADA , Swin Transformer , T5 , TD-MPC , TerraMind , TimeChat , TranceptionEVE , TreeRing , Tulu 2 , UnifiedQA , UniPerceiver , UNITER , UniVL , Van Gogh Diffusion , VERA , VGG / VGG13 , Vicuna , Video-Chat-7B , Video-ChatGPT , Video-LLaMA , Video-LLaMA-2-13B , Video-LLaVA , VideoCLIP , ViLA-8B , ViLBERT , VindLU , VioLET , ViRTex , ViT , ViV1T , VTG-LLM , WildGuard , Wizardlm , X-CLIP , X-InstructBLIP-7B , XGen-MM , Xlm-R , Zephyr-7B-beta Linear representation also in BLOOM , Cambrian-1 , Chameleon , CLIP / CLIP-ViT (LC) , DINOv2 , EVA-CLIP , Falcon , Gemma , Gemma 2 , GPT-2 , GPT-J , HPSv2 , ImageBind , InstructBLIP , LanguageBind , LLaMA , Llama 2 / Llama 2 base , Llama 3 , Llama 3.1 , Llama-3.2-3B , LLaVA-NeXT / LLaVA 1.6 , MAE , Mamba , Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1 , OLMo / OLMo base , OpenCLIP , Phi-3 , PickScore , Pythia , Qwen2-VL , Qwen2.5 , ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN , SALMONN , SAM , SigLIP , TerraMind , Tulu 2 , Vicuna , ViT Scale-dependent behaviour also in Aquila-7B , BEiT , BERT , BLOOM , Claude 2.1 , Claude 3 , Claude 3.5 , CLIP / CLIP-ViT (LC) , Code Llama , CodeGen , Cohere Command R , DeepSeek LLM , DeepSeekMoE , DeiT-III , DINO , DINOv2 , EquiformerV2 , ESCN , Falcon , FLAN-T5 , Gemini 1.0 Pro , Gemini 1.5 / Gemini Pro 1.5 , Gemma , Gemma 2 , GPT-2 , GPT-3 / GPT base , GPT-3.5 / ChatGPT-3.5 , GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report , GPT-4o , GPT-J , GPT-Neo , I3D , Idefics , InternLM-2.5-7B , InternLM-XComposer2-VL , InternLM2 , InternVL-1.5 , InternVL2 , LLaMA , Llama 2 / Llama 2 base , Llama 3 , Llama 3.1 , Llama 3.2 , Llama-3.2-3B , LLaVA-NeXT / LLaVA 1.6 , LongVA-7B , MAE , Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1 , Mixtral , Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b , Moirai , MPT , MViT V2 , O1 / OpenAI-o1-preview , OLMo / OLMo base , OpenCLIP , OpenFlamingo , OpenLLaMA , OPT , PaLM 2 , Phi-3 , Platypus2-Instruct-70B , Pythia , Qwen , Qwen1.5 , Qwen 2 , Qwen 2.5 72B Instruct , Qwen2-VL , Qwen-Audio , Qwen2.5 , RedPajama-INCITE , ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN , SlowFast , Solar 10.7B , Stable Diffusion , StableLM , Swin Transformer , T5 , TerraMind , text-ada-001 , TigerBot-70B-Chat , TimesFormer , TSM , Tulu 2 , Uniformer , Vicuna , VideoMAE , ViLA-8B , Wizardlm , X3D , XGLM , Xwin-LM-70B-v0.1 , Yi Shortcut also in BakLLaVA , BLIP-2 , Claude 3 , Claude 3.5 , CLIP / CLIP-ViT (LC) , CLIP4Clip , CLIPBERT , DALL·E 2 , DALL·E 3 , DeepSeek-VL2 , Eurus-RM-7B , Falcon , FLUX / FLUX1 , Gemini , Gemini 1.5 / Gemini Pro 1.5 , Gemma , Gemma 2 , GPT-2 , GPT-3 / GPT base , GPT-3.5 / ChatGPT-3.5 , GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report , GPT-4o , GPT-J , iFlytekSpark-13B , InstructBLIP , Internlm2-Reward , InternVideo , InternVL2 , LLaMA , Llama 2 / Llama 2 base , Llama 3 , Llama-3-2-Vision , LLaVA , LLaVA-Med , LLaVA-NeXT / LLaVA 1.6 , Med-Flamingo , Merlot Reserve , MiniGPT-4 , Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1 , Mixtral , Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b , Molmo , mPLUG-2 , mPLUG-Owl3 , Nova Canvas , O1 / OpenAI-o1-preview , OpenCLIP , OPT , Otter , PaLM 62B , Pythia , Qwen , Qwen 2 , Qwen-VL , Qwen2.5 , RadFM , ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN , Stable Diffusion , Swin Transformer , Sybil , TerraMind , Tulu 2 , UniPerceiver , UniVL , Vicuna , Video-LLaMA , VideoCLIP , VindLU , VioLET , X-CLIP