Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1

2023-10 · text · generative · anchor

Open-weight 7B language model, present in the corpus in its base and instruction-tuned releases at two point versions.

Variants

Mistral 7B v0.1, Mistral 7B v0.2, Mistral 7B Instruct v0.2, Mistral-7B-Instruct, Mistral-7B-Instruct-v0.1, Mistral-7B-Instruct-v0.3, HarmBench-Mistral, Mistral 7B v0.3, Mistral-Instruct-7B, Mistral Nemo Instruct 2407, Mistral-v0.3 7B

Findings

• IC-004Retrieval heads are sparse, universal, and causally responsible for long-context retrieval in LLMs
• IC-007Most LLMs do not align closely with human moral preferences on multilingual trolley problems
• IC-009LLM moral preferences show significant language sensitivity but not inequality toward low-resource languages
• IC-034Benefit and detriment in RAG can be traded off at token level for Llama-2, OPT and Mistral using representation similarity
• IC-043Five ~7B decoder-only LLMs develop a high-intrinsic-dimensionality phase in intermediate layers that marks the transition from surface-form to abstract linguistic processing, with earlier onset predicting better next-token prediction
• IC-074Released LLMs achieve F1 plan scores between 42.7 and 86.7 on the T-Eval plan task
• IC-1151LLaMA, OPT, LLaMA-2, Mistral, and GPT-J all exhibit token co-occurrence reinforcement, where the probability of generating a token increases monotonically with the number of its contextual co-occurrences
• IC-1549All 28 evaluated LMs exhibit gender bias on non-stereotypical sentence pairs, with fairness scores between 9% and 41%
• IC-1551No consistent relationship between model size and gender fairness scores is observed across six LM families
• IC-1576Base and aligned LLMs share 77.7% of top-1 token predictions, with distribution shifts concentrated in stylistic tokens rather than knowledge content
• IC-1577Base LLMs prompted with URiAL (3 restyled in-context examples + system prompt) match or surpass their SFT/RLHF-aligned counterparts on multi-aspect evaluation
• IC-166A 1-dimensional subspace in a single layer encodes the context-versus-prior decision in Llama-3.1-8B, Gemma-2 9B, and Mistral-v0.3 7B, and setting this subspace steers the released (non-fine-tuned) models' behavior
• IC-167Adding a PCA-derived control vector to the middle-layer residual stream improves logit-based reasoning accuracy on Pythia-1.4b, Pythia-2.8b, and Mistral-7B-Instruct
• IC-168Control vectors derived from BABI improve GSM8K accuracy and vice versa on Mistral-7B-Instruct, indicating a task-general reasoning direction in the residual stream
• IC-175Context-sufficiency performance is scale-dependent: larger LLMs achieve high accuracy with sufficient context but still answer correctly 35-62% of the time without it, while smaller models hallucinate or abstain even with sufficient context
• IC-181Truthfulness in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is linearly decodable from internal representations at exact answer tokens, with middle-to-late layers being most informative
• IC-182Truthfulness encoding in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is skill-specific rather than universal; probing classifiers do not meaningfully generalize across different task types beyond logit-based baselines
• IC-183Error types in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct are linearly predictable from internal representations, encoding fine-grained information beyond binary correctness
• IC-184Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct can internally encode the correct answer while externally generating an incorrect one, with the discrepancy most pronounced for error types where the model shows no external preference for the correct answer
• IC-198Safety-aligned LLMs (GPT-4, GPT-3.5, Gemma2-27b, GPT-4o, Gemma2-9b, Qwen2.5-72b, Mistral-7b, Mixtral-8x22b) are vulnerable to natural prompts semantically related to toxic seed prompts, with attack success rates of 82-99%
• IC-238LLMs fail to follow user preferences in zero-shot settings, with accuracy below 10% at 10 turns and near zero at 300 turns
• IC-239Implicit preference forms (choice-based and persona-driven) are significantly harder for LLMs to follow than explicit preferences at the same context length
• IC-240Introducing multiple preferences (including conflicting ones) in a conversation improves LLM adherence to the original preference
• IC-242Most LLMs exhibit higher bias ratios in multi-turn dialogues than in single-turn, with bias accumulating across successive turns
• IC-244No LLM demonstrates consistently strong fairness across both comprehension-focused and bias-resistance multi-turn tasks; models show complementary failure patterns
• IC-245Pretrained LLMs produce duration-dependent outputs that are incompatible with a discrete token interpretation
• IC-246Pretrained LLMs assign coherent semantic meaning to linear interpolations between token embeddings, extending the linear embedding hypothesis to the output space
• IC-247Pretrained LLMs are invariant to positional shifts but sensitive to duration scaling of the input
• IC-248Instruction fine-tuning causes context reliance under knowledge conflicts to initially increase then decrease (context-parametric inversion) in Llama2-7B, Pythia-6.9B, and Mistral-7B
• IC-264All 18 evaluated LLMs fail to abstain when the provided context lacks the answer, with performance gaps of 13.6% to 68.4% relative to the original context
• IC-265Model families show extreme variation in detecting conflicting answers in inconsistent contexts, with phi-3 series at 5.8% average accuracy versus GPT-4 series at 89.35%
• IC-266GPT-4o drops from 96.3% closed-book accuracy to 47.5% when given counterfactual context that contradicts its parametric knowledge, far below the 95% human accuracy on the same items
• IC-275Mistral 7B Instruct exhibits a reasoning-type-dependent failure mode where certain problems are exclusively solvable by one non-deductive reasoning type
• IC-304Instruction-tuned LMs become more vulnerable to prompt-injected data extraction as model size increases from 7B to 70B
• IC-305Mistral-instruct-7b's susceptibility to prompt-injected data extraction follows a U-shaped curve depending on the position of the adversarial prompt within the context window
• IC-306Instruction tuning increases the ROUGE score of prompt-injected data extraction by 65.76 on average compared to base models
• IC-30756 LLMs on Sorry-Bench show fulfillment rates ranging from below 10% (Claude-2, Gemini-1.5) to above 90% (Mistral-7B-instruct-v0.1, Dolphin-2.6-mixtral-8x7b), with GPT-4o at 30% and Llama-3-70B at 35%
• IC-308Linguistic mutations to unsafe prompts significantly and inconsistently alter safety refusal across models, with persuasion techniques increasing fulfillment by 5-66% and encoding/encryption decreasing it by 15-68%
• IC-309As zero-shot safety judges, GPT-4o achieves 78.9% Cohen's kappa agreement with human annotators while Llama-3-8B-instruct (39.0%) and Mistral-7B-instruct-v0.2 (53.9%) perform substantially worse
• IC-327Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and several other LLMs produce well-calibrated verbal confidence estimates on classification tasks
• IC-328Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.3 are susceptible to confidence-elicitation-guided word substitution attacks, with CEAttack outperforming existing hard-label black-box methods
• IC-340GCG jailbreaking attacks exhibit strong model-specific transferability, achieving below 3% ASR on Llama-2-13b-chat and Llama-3.1-8b-instruct but above 90% ASR on Vicuna-13b-v1.5 and Mistral-7b-instruct
• IC-358GPT-2-small and Mistral 7B contain circular representations of days of the week and months of the year in their internal activations, discovered via SAE dictionary element clustering
• IC-359Mistral 7B and Llama 3 8B causally use circular subspaces to compute modular arithmetic on days of the week and months of the year
• IC-361Mistral 7B's circular representation of days of the week is continuous, mapping intermediate time-of-day values to positions between adjacent weekdays
• IC-36570B LLM variants tolerate substantially higher activation sparsity than smaller counterparts, and Llama-3 shows more degradation than Llama-2 and Mistral at 50% sparsity
• IC-395Most mainstream LLMs exhibit positive ADCE across five tasks, indicating reliance on deep structure for problem-solving, with ADCE strongly correlated with accuracy (r² > 0.7)
• IC-396Closed-source LLMs (GPT, Claude) rely more on deep structure than open-source LLMs (Llama, Mistral), and open-source models' surface sensitivity decreases with model scale
• IC-397Mistral 7B Instruct and Llama 3 8B Instruct exhibit systematic misalignment between their operational semantics of subjective phrases and human expectations, producing unexpected side effects when steered with certain phrases
• IC-402In LLaMA3-8B, LLaMA2-13B, and Mistral-7B, soft-prompt information flow peaks in shallow layers (2–10) and reasoning correctness depends on whether deeper layers redirect attention away from soft prompts to earlier reasoning steps
• IC-409Knowledge editing methods correct verified hallucinations in Llama2-7B, Llama3-8B, and Mistral-v0.3-7B far less effectively than their scores on existing benchmarks suggest
• IC-410Knowledge editing can degrade generalization performance below pre-edit levels in Llama2-7B, Llama3-8B, and Mistral-v0.3-7B
• IC-411Llama2-7B, Llama3-8B, and Mistral-v0.3-7B do not reason with edited knowledge in multi-hop questions, as editing methods mostly underperform pre-edit portability scores
• IC-412Edited knowledge in Llama2-7B is significantly less robust to adversarial prompts than in Llama3-8B and Mistral-v0.3-7B
• IC-421Sequential context-switching queries jailbreak Llama and Mistral models at 95% attack success rate
• IC-435Mistral-7B employs a less efficient algorithmic strategy (O(n²)) than Llama-3-8B (O([n², n³])) on probe tasks with multiple solution complexities
• IC-439Agent-specialized fine-tuned models (XLAM) significantly improve API selection over base models, but code-fine-tuned models (AgentLM) degrade performance, and no fine-tuning approach improves input recognition
• IC-451Transformer block coupling of Jacobian singular vectors positively correlates with benchmark performance across 30+ LLMs, more strongly than parameter count, depth, or embedding dimension
• IC-461Adding calibrated Gaussian noise to steering vectors (PSA) preserves alignment performance comparable to non-private mean steering across Llama-2-7B, Mistral-7B, Gemma-2-2B, and Qwen-2.5-7B
• IC-485LLMs show a significant performance gap between Wikipedia-based factual multi-hop QA and counterfactual multi-hop QA, indicating reliance on memorized knowledge rather than reasoning from context
• IC-493A linear direction in the input embedding space of Llama-2-7B-Chat, Llama-2-13B-Chat, Mistral-7B-Instruct-v0.3, and Phi-3-mini-128k predicts instruction-following success, generalizes across tasks but not instruction types, and can be used to improve adherence via representation engineering
• IC-501Linear probes on middle-layer attention heads of Llama-2-7B-Chat, Mistral-7B-Instruct-v0.1, and Vicuna-7B-v1.5 predict US lawmakers' DW-Nominate ideology scores with Spearman correlations around 0.85
• IC-502Linear probes trained on US lawmaker ideology generalize to predict Ad Fontes media slant scores when the same models simulate news outlets
• IC-503Adding probe regression coefficients to attention head activations steers Llama-2-7B-Chat, Mistral-7B-Instruct-v0.1, and Vicuna-7B-v1.5 toward more liberal or conservative generated text
• IC-505Adversarial attacks on Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2, and Gemma-7B-IT shift hidden representations along the negative refusal feature direction
• IC-506Restoring the refusal feature in Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2, and Gemma-7B-IT causally disables all four tested adversarial attacks
• IC-507The refusal feature direction in Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2, and Gemma-7B-IT ranks near the top among 100 perturbations for compromising model safety
• IC-511LLaMA-2, Gemma, and Mistral all perform in-context density estimation via an adaptive kernel-like process, as revealed by their similar low-dimensional INPCA trajectories bounded between the geodesic and the Gaussian submanifold
• IC-549All 18 evaluated LLMs show a 15-20% performance gap between linear (node chain) and graph (workflow) planning on WorfBench
• IC-550Workflow generation performance scales with model size within families, but recently released 7B models outperform older 13B models
• IC-556All 9 LLM-based guard models exhibit significant miscalibration with average ECE exceeding 10% across 12 public benchmarks for both prompt and response classification
• IC-557Guard models show significantly degraded calibration under jailbreak attacks, with prompt classification ECE substantially higher than response classification ECE
• IC-558Guard models exhibit inconsistent calibration when classifying responses from different response model types, with ECE varying by up to 39 percentage points within a single model
• IC-559Contextual calibration is most effective for prompt classification while temperature scaling is more effective for response classification, but no single post-hoc method fully resolves miscalibration
• IC-566Adding instruction-specific steering vectors to the residual stream improves instruction-following accuracy for Phi-3, Gemma 2 2B IT, Mistral 7B IT, and Gemma 2 9B IT across format, length, and word-specific constraints
• IC-569Mistral-7B-instruct-v0.1 achieves only F1 of 0.419 on zero-shot stance detection for the X-Stance German dataset, substantially below the fine-tuned BERT baseline (F1 0.693)
• IC-575Four released LLMs (LLaMA-3.1-8B, Mistral-7B, Qwen2-7B, Yi-1.5-9B) can perform in-context learning on continuous vector representations projected into their embedding space, matching or outperforming few-shot ICL across text, time-series, graph, and fMRI tasks
• IC-576For 10-digit numerical function regression, vector-ICL consistently outperforms few-shot ICL with raw number inputs across all four LLMs because continuous representations avoid multi-token splitting
• IC-590Tuning only the identified safety neurons (SN-Tune) reduces harmful scores of instruction-tuned and base models by over 90 points while preserving general capability.
• IC-591Downstream fine-tuning on GSM8K degrades safety of Llama2-7b-chat and Mistral-7b-instruct-v0.2, but RSN-Tune partially preserves safety by protecting non-overlapping safety neurons.
• IC-592The log-likelihood layer in LLaMA-2-7B, LLaMA-2-7B-Chat, Vicuna-7B, and Mistral-7B-Instruct produces factually incorrect answers on TruthfulQA MC1 (817 samples) due to a misalignment between the output distribution and internal attention head representations, with LM-to-head-norm accuracy gaps of 24.23 to 40.68 points.
• IC-593The L2 norms of attention heads in Mistral-7B-Instruct and LLaMA-2-7B correlate with truthfulness, spiking by up to 83% at token positions of factual proposition completions and pertinent factual associations, and this correlation is specific to multi-headed attention representations rather than query, key, value, output, or FFN norms.
• IC-594In LLaMA-2-7B, the truth-correlated attention heads are concentrated after layer 9, with two functional types (structural and associative) evenly distributed throughout the upper portions of the model, showing no further depth-dependent specialisation within that region.
• IC-596Multiple released LLMs fail to refuse harmful prompts disguised as historical or philosophical discussions, with GPT-4o and Mixtral showing the lowest refusal rates
• IC-608Token trajectories in GPT-2, Llama 2 7B, Mistral 7B, and Llama 3.2 models cluster on a low-dimensional manifold and follow a linear drift plus Gaussian noise dynamics
• IC-609The last transformer layer of Mistral 7B, Llama 3.2 1B, and Llama 3.2 3B shows anomalous trajectory statistics inconsistent with the linear drift-plus-noise pattern of intermediate layers
• IC-868Skill-Mix performance degrades with increasing k, and within the Llama-2 family the saturation point increases with model size
• IC-869Models ranking highly on popular LLM leaderboards perform worse than Llama-2-70b-chat on Skill-Mix, suggesting cramming for the leaderboard at the expense of general-purpose text generation
• IC-871Llama-2-70b-chat as a grader is more generous than GPT-4 and systematically gives higher scores to Llama-2 family outputs

Shared mechanisms

• Circular representationalso in Gemini 1.0 Pro, Gemma 2, GPT-2, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, Llama 3, Llama 3.1, TerraMind
• Depth-dependent structurealso in Baichuan 2, BERT, BLIP-2, BLOOM, Chameleon, CLIP / CLIP-ViT (LC), DeepFloyd IF, DeiT-III, DINO, DINOv2, Falcon, Gemma, Gemma 2, GPT-2, GPT-J, GPT-NeoX-20B, Griffin, I3D, Idefics, InstructBLIP, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Phi, MAE, MAE-B/16, Mamba, MiniGPT-4, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, mPLUG-Owl, MPT, MultiBERTs, MViT V2, OLMo / OLMo base, OpenCLIP, OPT, Phi-2, Pythia, Qwen 2, Qwen2-VL, Qwen2.5, RoBERTa / RoBERTa-L, RWKV, SALMONN, SAM, SlowFast, Stable Diffusion, Swin Transformer, TerraMind, TimesFormer, TSM, Uniformer, Vicuna, VideoMAE, ViT, X3D, Yi
• Failure modealso in AASIST, ADM, Aegis-Guard-Defensive, Alpaca, AnyLoc, AutoTikZ / DataTikZ, Baichuan, Baichuan 2, Baichuan2-13B, BakLLaVA, Bard, BEiT, BERT, BingChat, BLIP, BLIP-2, BLOOM, BSRNet, CF2, Chat-UniVi-7B, ChatGLM-6B / ChatGLM-6b-2, ChatGLM2, ChatGPT, CLAP, Claude 1.3, Claude 2.0, Claude 2.1, Claude 3, Claude 3.5, CLEAR, CLIP / CLIP-ViT (LC), CLIP4Clip, CLIPBERT, CLIPCap, CLMBR-T-BASE, CloFNet, Code Llama, CodeGeex2, CodeGen, CodeLlama-13B, CodeLlama-34B, CogVLM2, Cohere Command R, CoMEt, Command R+, CONCH, CycleGAN, DALL-E, DALL·E 2, DALL·E 3, DASR, DECAF, DeepSeek-2-Chat, DeepSeek-2-Coder, DeepSeek-V2-0628, DeepSeek-VL, DeepSeek-VL2, DeiT, DeiT-III, Depth Anything, DETR-R50, DimeNet++, DINO, DINO-FocalNet-Large, DINOv2, EGNN, Emu2, EquiformerV2, ESCN, ESM-2, ESM3, ESRGAN, EVA-CLIP, EVE, Falcon, Faster R-CNN / Faster R-CNN R50 / Faster R-CNN X101, FLAN-T5, Florence-2, FLUX / FLUX1, FoveaBox, Fuyu, Galactica-6.7B, GAT, 3D Gaussian Splatting, GCN, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GIN, GLIDE, GLM-4, GLM-4V, GloVe, GP-UNIT, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4.1, GPT-4o, GPT-J, GPT-NeoX-20B, GraphSAGE, Grounding DINO, Guanaco, GVP, Hawkeye, HiFaceGAN, HPS, HuggingGPT, IDDPM, Idefics, Idefics2, ImageBind, ImageBind-LLM-7B, Imagen Video, ImageReward, 12-in-1, InstructBLIP, InstructGPT, InternLM-2.5-7B, InternLM-XComposer2-VL, InternVideo, InternVL-1.5, InternVL2, Koala, LegalBERT, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, Llama-3-2-Vision, LLaMA-Adapter v2, Llama Guard, Llama Guard 2, Llama-Guard 3, Llama-VID, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, LLaVA-OneVision, LongVA-7B, LOVT, LWM-1M-JAX, MACE, MAE, Med-Flamingo, Merlot Reserve, MGCA, MiDaS, MiniCPM-V, MiniGPT-4, Mip-Splatting, Mistral Large 2, Mistral Large V2, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, MobileNetV2, Molmo, MolmoE-7B, Momentor, Moondream2, Moonshot-v1-8k, mPLUG-2, mPLUG-Owl, mPLUG-Owl3, mPLUG-Owl2, MPT, MSA Transformer, MultiBERTs, Nova Canvas, Nova Lite, Nova Pro, O1 / OpenAI-o1-preview, O3, O4-mini, OLMo / OLMo base, OneLLM, OpenAI Moderation, OpenChat-3.5-0106, OpenCLIP, OpenFlamingo, OPT, Otter, Otter-7B, PaLM 2, PaLM 62B, PandaGPT-7B, PerSAM, Phi-3, Phi-3.5 Mini Instruct, PickScore, PLIP, Prismatic, ProGen-2, Pythia, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen-VL, Qwen2.5, Qwen2-Audio, R2D2, RadFM, RCExplainer, RD-SR, RealESRNet, Reprover, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, RetinaNet-R50, RivaGAN, RS-LDS, RT-1, RT-2, SALMONN, SAM, SAM 2, SAULLM 54B, Scaffold-GS, SchNet, Seed-LLaMA-8B, SGC, SIREN, Sketch Transformer, SLD-max, SLD-medium, SLD-strong, SLDS, SLIMG, SpeechGPT, SphereNet, SRResNet, Stable Diffusion, StackLLaMA, Starcoder, StegaStamp, StyleGAN2-ADA, Swin Transformer, T5, TD-MPC, TerraMind, TimeChat, TranceptionEVE, TreeRing, Tulu 2, UnifiedQA, UniPerceiver, UNITER, UniVL, Van Gogh Diffusion, VERA, VGG / VGG13, Vicuna, Video-Chat-7B, Video-ChatGPT, Video-LLaMA, Video-LLaMA-2-13B, Video-LLaVA, VideoCLIP, ViLA-8B, ViLBERT, VindLU, VioLET, ViRTex, ViT, ViV1T, VTG-LLM, WildGuard, Wizardlm, X-CLIP, X-InstructBLIP-7B, XGen-MM, Xlm-R, Zephyr-7B-beta
• Linear representationalso in BLOOM, Cambrian-1, Chameleon, CLIP / CLIP-ViT (LC), DINOv2, EVA-CLIP, Falcon, Gemma, Gemma 2, GPT-2, GPT-J, HPSv2, ImageBind, InstructBLIP, LanguageBind, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, MAE, Mamba, OLMo / OLMo base, OpenCLIP, Phi-3, PickScore, Pythia, Qwen2-VL, Qwen2.5, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SALMONN, SAM, SigLIP, TerraMind, Tulu 2, Vicuna, ViT
• Method artefactalso in Baichuan, CLIP / CLIP-ViT (LC), ConvNeXt, EfficientNet, Falcon, Gemma, Gemma 2, GPT-2, GPT-3.5 / ChatGPT-3.5, GPT-4o, GPT-J, InternLM, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, MAP-NEO, OLMo / OLMo base, OpenCLIP, OpenLLaMA, OPT, Pythia, Qwen1.5, Qwen 2, Qwen2.5, RedPajama, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, Skywork, Stable Diffusion, StableLM, TerraMind, ViT, Yi, ZiYA2
• Positional biasalso in BERT, ChatGPT, Claude 3, Claude 3.5, Falcon, Fuyu, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GLIDE, GPT-2, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4.1, GPT-4o, GPT-J, InstructGPT, LLaMA, Llama 2 / Llama 2 base, Llama 3, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, MPT, O1 / OpenAI-o1-preview, O3, O4-mini, PaLM 2, Phi-3, Pythia, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Stable Diffusion, Sybil, Vicuna
• Scale-dependent behaviouralso in Aquila-7B, BEiT, BERT, BLOOM, Claude 2.1, Claude 3, Claude 3.5, CLIP / CLIP-ViT (LC), Code Llama, CodeGen, Cohere Command R, DeepSeek LLM, DeepSeekMoE, DeiT-III, DINO, DINOv2, EquiformerV2, ESCN, Falcon, FLAN-T5, Gemini 1.0 Pro, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, GPT-Neo, I3D, Idefics, InternLM-2.5-7B, InternLM-XComposer2-VL, InternLM2, InternVL-1.5, InternVL2, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, LongVA-7B, MAE, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Moirai, MPT, MViT V2, O1 / OpenAI-o1-preview, OLMo / OLMo base, OpenCLIP, OpenFlamingo, OpenLLaMA, OPT, PaLM 2, Phi-3, Platypus2-Instruct-70B, Pythia, Qwen, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen2.5, RedPajama-INCITE, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SlowFast, Solar 10.7B, Stable Diffusion, StableLM, Swin Transformer, T5, TerraMind, text-ada-001, TigerBot-70B-Chat, TimesFormer, TSM, Tulu 2, Uniformer, Vicuna, VideoMAE, ViLA-8B, Wizardlm, X3D, XGLM, Xwin-LM-70B-v0.1, Yi
• Shortcutalso in BakLLaVA, BLIP-2, Claude 3, Claude 3.5, CLIP / CLIP-ViT (LC), CLIP4Clip, CLIPBERT, DALL·E 2, DALL·E 3, DeepSeek-VL2, Eurus-RM-7B, Falcon, FLUX / FLUX1, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, iFlytekSpark-13B, InstructBLIP, Internlm2-Reward, InternVideo, InternVL2, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama-3-2-Vision, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, Med-Flamingo, Merlot Reserve, MiniGPT-4, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Molmo, mPLUG-2, mPLUG-Owl3, Nova Canvas, O1 / OpenAI-o1-preview, OpenCLIP, OPT, Otter, PaLM 62B, Pythia, Qwen, Qwen 2, Qwen-VL, Qwen2.5, RadFM, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, Stable Diffusion, Swin Transformer, Sybil, TerraMind, Tulu 2, UniPerceiver, UniVL, Vicuna, Video-LLaMA, VideoCLIP, VindLU, VioLET, X-CLIP