Claude 3.5

Anthropic · text, image · generative · anchor

Closed-weight multimodal model family served through an API, cited in the corpus at the Sonnet tier.

Note

closed-weight model with no paper; the anchor is the vendor announcement for the Sonnet tier carried here

Variants

Claude 3.5 Sonnet, Claude 3.5 Haiku, Claude-3.5-Sonnet-20240620

Findings

• IC-001Vision-language models perform near chance on the NL-Eye visual abductive reasoning benchmark
• IC-002Even when VLMs select the correct hypothesis, their explanations are often invalid or unhelpful
• IC-072LLM agents of varying scales exhibit a failure mode on web automation tasks when processing raw, complex web page observations, with the penalty being more severe for smaller models
• IC-082GPT-3.5, GPT-4o, Claude-3.5-Sonnet, and Llama-3.1-8B produce explanations on the BBQ social bias task that are systematically unfaithful for identity and behavior concepts while remaining faithful for context concepts, with specific patterns of hiding safety-measure influence and social bias
• IC-083GPT-3.5, GPT-4o, and Claude-3.5-Sonnet produce unfaithful explanations on MedQA medical questions, omitting high-effect clinical concepts such as the patient's mental status while over-referencing low-effect concepts
• IC-092All evaluated LLMs show consistent F1 degradation to at most 0.60 when two or more events match a retrieval cue
• IC-093No evaluated LLM achieves perfect confabulation avoidance on questions about non-existent events
• IC-094Episodic recall accuracy degrades systematically from content cues to space cues to time cues across all evaluated LLMs
• IC-095Evaluated LLMs achieve at most 36% latest-state accuracy and 18% full-set accuracy on multi-event entity tracking, with low Kendall's tau on chronological ordering
• IC-170GPT-4, GPT-3.5, and Claude-3.5-Sonnet rely heavily on parametric knowledge in RAG settings, producing ungrounded responses with high answered ratios and low trust-scores
• IC-171ICL prompting produces binary response patterns in released LLMs, with answered ratios collapsing to near 0% or 100% rather than calibrated refusal, making prompting ineffective for RAG groundedness
• IC-174RAG reduces model abstention and LLMs hallucinate rather than abstain when the retrieved context is insufficient to answer the query
• IC-175Context-sufficiency performance is scale-dependent: larger LLMs achieve high accuracy with sufficient context but still answer correctly 35-62% of the time without it, while smaller models hallucinate or abstain even with sufficient context
• IC-202All six evaluated LLMs achieve very low accuracy on OpenRCA, with no model solving any three-element root cause query
• IC-203Gemini 1.5 Pro's RCA-Agent accuracy drops 68.4% when code execution fails, far exceeding the drops for Claude 3.5 (17.9%) and GPT-4o (15.6%)
• IC-238LLMs fail to follow user preferences in zero-shot settings, with accuracy below 10% at 10 turns and near zero at 300 turns
• IC-255GPT-4o, Llama 3.1, and Claude models show varying correlation with human ratings when used as stereotype evaluators, with GPT-4o achieving the strongest gender correlation (ρ=0.86) but weaker racial correlations
• IC-264All 18 evaluated LLMs fail to abstain when the provided context lacks the answer, with performance gaps of 13.6% to 68.4% relative to the original context
• IC-265Model families show extreme variation in detecting conflicting answers in inconsistent contexts, with phi-3 series at 5.8% average accuracy versus GPT-4 series at 89.35%
• IC-266GPT-4o drops from 96.3% closed-book accuracy to 47.5% when given counterfactual context that contradicts its parametric knowledge, far below the 95% human accuracy on the same items
• IC-267Adding a 'conflict' instruction to the prompt degrades GPT-4o and Claude 3.5 Sonnet accuracy on normal (answerable, consistent) contexts by 5% and 2% respectively
• IC-276All 14 evaluated VLMs show a large gap between average-case and worst-case accuracy on DynaMath variants, with worst-case at or below 50% of average-case, and the failures are systematic rather than random
• IC-278Claude-3.5 Sonnet and GPT-4o exhibit a memorization failure mode, outputting the same answer regardless of visual parameter changes in the problem
• IC-322Past-tense reformulations of harmful requests bypass refusal training in eight released LLMs, while future-tense reformulations are substantially less effective
• IC-343GPT-4o, Claude-3.5 Sonnet, and GeminiPro-1.5 score below BigDocs-trained open models on BigDocs-Bench tasks requiring long structured code generation
• IC-344GPT-4o achieves the highest average score (64.62) on general document benchmarks, outperforming Qwen2-VL-72B (58.40) and GeminiPro-1.5 (57.05)
• IC-368Larger LMs (GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro) exhibit better calibration than their smaller counterparts (GPT-4o-mini, Claude-3-Haiku, Gemini-1.5-Flash) when verbalizing confidence with certainty phrases
• IC-369LMs verbalizing confidence with certainty phrases are better calibrated on SCIQ than on TruthfulQA
• IC-395Most mainstream LLMs exhibit positive ADCE across five tasks, indicating reliance on deep structure for problem-solving, with ADCE strongly correlated with accuracy (r² > 0.7)
• IC-396Closed-source LLMs (GPT, Claude) rely more on deep structure than open-source LLMs (Llama, Mistral), and open-source models' surface sensitivity decreases with model scale
• IC-408Claude models (2.0, 2.1, 3 Haiku, 3 Sonnet, 3 Opus, 3.5 Sonnet) achieve 100% jailbreak attack success rate under prefilling attacks via the Anthropic API
• IC-440GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet show up to 25% skill-level accuracy gaps despite overall accuracies within 0.4% of each other
• IC-441Skill-level improvements between model releases are highly uneven, with Claude 3.5 Sonnet gaining ~50% over Claude 3 Opus on law skills while Gemini improved most in math and science
• IC-442Routing each evaluation instance to the model strongest on its relevant skills yields a 3.2% accuracy gain over the best single model, with 3.5-6.8% gains on MMLU Pro
• IC-443Model inconsistency on probing questions negatively correlates with skill-slice accuracy (r = -0.675), with models contradicting themselves more often on skills where they perform poorly
• IC-500GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1 405B solve multi-step retrieval problems without fine-tuning, achieving near-perfect accuracy for chains of up to 5 steps
• IC-518GPT-4o achieves 62.54% overall accuracy on MMWorld, the best among 15 MLLMs, while four open-source models perform below the 26.31% random-choice baseline
• IC-519MLLMs exhibit different skill sets than humans, correctly answering expert-level questions that all three human annotators miss while failing on easy questions humans answer correctly
• IC-520Temporal reasoning performance drops significantly across all 15 MLLMs when video frames are shuffled or reduced to one-fifth of the original count
• IC-522LLMs perform correct example inference without inducing the correct rule, and this gap is robust to prompting methods, fact count, and scenario form
• IC-523LLMs rely on observed facts close to the test case in input feature space (neighbor-based reasoning) rather than on an abstract rule, and this effect is localized
• IC-549All 18 evaluated LLMs show a 15-20% performance gap between linear (node chain) and graph (workflow) planning on WorfBench
• IC-572Bijection learning achieves state-of-the-art jailbreak ASR on frontier models, with peak ASR increasing with model capability
• IC-573Model capabilities on MMLU degrade monotonically as bijection encoding complexity increases
• IC-574Guard models fail to effectively mitigate bijection attacks even at capability parity with the target model
• IC-597Most LMMs exhibit systematic class bias in synthetic data detection, with GPT-4o biased toward classifying text as real and 3D as AI-generated
• IC-600Chain-of-thought prompting improves most LMMs on synthetic detection but degrades LLaVA-ov-7b from 56.6% to 18.8%, while GPT-4o performs well without it (64.1% baseline)

Shared mechanisms

• Explanation faithfulnessalso in BakLLaVA, CF2, Claude 3, CLIP / CLIP-ViT (LC), DRUM, Fuyu, GEM, Gemini 1.5 / Gemini Pro 1.5, Gemma 2, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, Idefics, Llama 3.1, LLaVA-NeXT / LLaVA 1.6, MAE-B/16, MobileNetV2, mPLUG-Owl3, OpenFlamingo, PGExplainer, Pythia, RCExplainer, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SigLIP, SigLIP-2, Stable Diffusion, TAGExplainer, ViT
• Failure modealso in AASIST, ADM, Aegis-Guard-Defensive, Alpaca, AnyLoc, AutoTikZ / DataTikZ, Baichuan, Baichuan 2, Baichuan2-13B, BakLLaVA, Bard, BEiT, BERT, BingChat, BLIP, BLIP-2, BLOOM, BSRNet, CF2, Chat-UniVi-7B, ChatGLM-6B / ChatGLM-6b-2, ChatGLM2, ChatGPT, CLAP, Claude 1.3, Claude 2.0, Claude 2.1, Claude 3, CLEAR, CLIP / CLIP-ViT (LC), CLIP4Clip, CLIPBERT, CLIPCap, CLMBR-T-BASE, CloFNet, Code Llama, CodeGeex2, CodeGen, CodeLlama-13B, CodeLlama-34B, CogVLM2, Cohere Command R, CoMEt, Command R+, CONCH, CycleGAN, DALL-E, DALL·E 2, DALL·E 3, DASR, DECAF, DeepSeek-2-Chat, DeepSeek-2-Coder, DeepSeek-V2-0628, DeepSeek-VL, DeepSeek-VL2, DeiT, DeiT-III, Depth Anything, DETR-R50, DimeNet++, DINO, DINO-FocalNet-Large, DINOv2, EGNN, Emu2, EquiformerV2, ESCN, ESM-2, ESM3, ESRGAN, EVA-CLIP, EVE, Falcon, Faster R-CNN / Faster R-CNN R50 / Faster R-CNN X101, FLAN-T5, Florence-2, FLUX / FLUX1, FoveaBox, Fuyu, Galactica-6.7B, GAT, 3D Gaussian Splatting, GCN, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GIN, GLIDE, GLM-4, GLM-4V, GloVe, GP-UNIT, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4.1, GPT-4o, GPT-J, GPT-NeoX-20B, GraphSAGE, Grounding DINO, Guanaco, GVP, Hawkeye, HiFaceGAN, HPS, HuggingGPT, IDDPM, Idefics, Idefics2, ImageBind, ImageBind-LLM-7B, Imagen Video, ImageReward, 12-in-1, InstructBLIP, InstructGPT, InternLM-2.5-7B, InternLM-XComposer2-VL, InternVideo, InternVL-1.5, InternVL2, Koala, LegalBERT, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, Llama-3-2-Vision, LLaMA-Adapter v2, Llama Guard, Llama Guard 2, Llama-Guard 3, Llama-VID, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, LLaVA-OneVision, LongVA-7B, LOVT, LWM-1M-JAX, MACE, MAE, Med-Flamingo, Merlot Reserve, MGCA, MiDaS, MiniCPM-V, MiniGPT-4, Mip-Splatting, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mistral Large 2, Mistral Large V2, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, MobileNetV2, Molmo, MolmoE-7B, Momentor, Moondream2, Moonshot-v1-8k, mPLUG-2, mPLUG-Owl, mPLUG-Owl3, mPLUG-Owl2, MPT, MSA Transformer, MultiBERTs, Nova Canvas, Nova Lite, Nova Pro, O1 / OpenAI-o1-preview, O3, O4-mini, OLMo / OLMo base, OneLLM, OpenAI Moderation, OpenChat-3.5-0106, OpenCLIP, OpenFlamingo, OPT, Otter, Otter-7B, PaLM 2, PaLM 62B, PandaGPT-7B, PerSAM, Phi-3, Phi-3.5 Mini Instruct, PickScore, PLIP, Prismatic, ProGen-2, Pythia, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen-VL, Qwen2.5, Qwen2-Audio, R2D2, RadFM, RCExplainer, RD-SR, RealESRNet, Reprover, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, RetinaNet-R50, RivaGAN, RS-LDS, RT-1, RT-2, SALMONN, SAM, SAM 2, SAULLM 54B, Scaffold-GS, SchNet, Seed-LLaMA-8B, SGC, SIREN, Sketch Transformer, SLD-max, SLD-medium, SLD-strong, SLDS, SLIMG, SpeechGPT, SphereNet, SRResNet, Stable Diffusion, StackLLaMA, Starcoder, StegaStamp, StyleGAN2-ADA, Swin Transformer, T5, TD-MPC, TerraMind, TimeChat, TranceptionEVE, TreeRing, Tulu 2, UnifiedQA, UniPerceiver, UNITER, UniVL, Van Gogh Diffusion, VERA, VGG / VGG13, Vicuna, Video-Chat-7B, Video-ChatGPT, Video-LLaMA, Video-LLaMA-2-13B, Video-LLaVA, VideoCLIP, ViLA-8B, ViLBERT, VindLU, VioLET, ViRTex, ViT, ViV1T, VTG-LLM, WildGuard, Wizardlm, X-CLIP, X-InstructBLIP-7B, XGen-MM, Xlm-R, Zephyr-7B-beta
• Positional biasalso in BERT, ChatGPT, Claude 3, Falcon, Fuyu, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GLIDE, GPT-2, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4.1, GPT-4o, GPT-J, InstructGPT, LLaMA, Llama 2 / Llama 2 base, Llama 3, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, MPT, O1 / OpenAI-o1-preview, O3, O4-mini, PaLM 2, Phi-3, Pythia, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Stable Diffusion, Sybil, Vicuna
• Scale-dependent behaviouralso in Aquila-7B, BEiT, BERT, BLOOM, Claude 2.1, Claude 3, CLIP / CLIP-ViT (LC), Code Llama, CodeGen, Cohere Command R, DeepSeek LLM, DeepSeekMoE, DeiT-III, DINO, DINOv2, EquiformerV2, ESCN, Falcon, FLAN-T5, Gemini 1.0 Pro, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, GPT-Neo, I3D, Idefics, InternLM-2.5-7B, InternLM-XComposer2-VL, InternLM2, InternVL-1.5, InternVL2, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, LongVA-7B, MAE, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Moirai, MPT, MViT V2, O1 / OpenAI-o1-preview, OLMo / OLMo base, OpenCLIP, OpenFlamingo, OpenLLaMA, OPT, PaLM 2, Phi-3, Platypus2-Instruct-70B, Pythia, Qwen, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen2.5, RedPajama-INCITE, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SlowFast, Solar 10.7B, Stable Diffusion, StableLM, Swin Transformer, T5, TerraMind, text-ada-001, TigerBot-70B-Chat, TimesFormer, TSM, Tulu 2, Uniformer, Vicuna, VideoMAE, ViLA-8B, Wizardlm, X3D, XGLM, Xwin-LM-70B-v0.1, Yi
• Shortcutalso in BakLLaVA, BLIP-2, Claude 3, CLIP / CLIP-ViT (LC), CLIP4Clip, CLIPBERT, DALL·E 2, DALL·E 3, DeepSeek-VL2, Eurus-RM-7B, Falcon, FLUX / FLUX1, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, iFlytekSpark-13B, InstructBLIP, Internlm2-Reward, InternVideo, InternVL2, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama-3-2-Vision, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, Med-Flamingo, Merlot Reserve, MiniGPT-4, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Molmo, mPLUG-2, mPLUG-Owl3, Nova Canvas, O1 / OpenAI-o1-preview, OpenCLIP, OPT, Otter, PaLM 62B, Pythia, Qwen, Qwen 2, Qwen-VL, Qwen2.5, RadFM, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, Stable Diffusion, Swin Transformer, Sybil, TerraMind, Tulu 2, UniPerceiver, UniVL, Vicuna, Video-LLaMA, VideoCLIP, VindLU, VioLET, X-CLIP