LLaMA

anchor

Variants

Llama 7B, LLaMA-13B, LLaMA-30B, LLaMA-65B

Findings

• IC-1053Instruction tuning suppresses in-context learning in LLaMA, Vicuna, and OPT-IML, with the suppression being largest for English prompts and partially recoverable via translation to other languages
• IC-1078LLaMA models (7B through 65B) exhibit gender bias in language generation, coreference resolution, and sentence likelihood, with stereotypical associations driving predictions
• IC-1079Causal tracing reveals that mid-upper MLP layers (layers 18–25 in 7B) are the primary mediators of stereotypical gender bias in LLaMA, while the last layers show negative coefficients that counter the bias
• IC-1088ICL predictions in LLaMA, LLaMA-2, and Falcon models depend on in-context label information and can learn truly novel label relationships
• IC-1089ICL in LLaMA, LLaMA-2, and Falcon models cannot fully overcome pre-training label preferences when in-context labels are flipped
• IC-1090ICL in LLaMA, LLaMA-2, and Falcon models preferentially uses in-context label information closer to the query rather than treating all examples equally
• IC-1151LLaMA, OPT, LLaMA-2, Mistral, and GPT-J all exhibit token co-occurrence reinforcement, where the probability of generating a token increases monotonically with the number of its contextual co-occurrences
• IC-1152Token reinforcement in demonstrations constrains LLaMA-65B's output to valid label spaces on MMLU and enables chain-of-thought pattern following on GSM8K without requiring question content
• IC-1153Intentionally constructed spurious token connections in MMLU demonstrations misdirect LLaMA-65B's in-context learning toward specific answer choices
• IC-1154LLaMA-65B exhibits a selection bias where zero-shot accuracy varies substantially by answer choice, with 'a' at 71.58% and 'd' at 52.28%
• IC-1217LLaMA 65B's token-probability readout fails to capture human decision-making, producing near-chance NLL and no human-like exploration behavior
• IC-1319Larger LLaMA and LLaMA2 models show better calibration on phrase-level tasks but not consistently on sentence- and paragraph-level tasks
• IC-1320GPT-2 XL (1.5B) exhibits better calibration than larger models from the LLaMA, LLaMA2, and GPT-J families despite having fewer parameters
• IC-1321Vicuna-13B, instruction-tuned from LLaMA-13B on user conversations, exhibits worse calibration than its base model LLaMA-13B
• IC-1386Fact recall in OPT and LLaMA models degrades by more than 5% relative accuracy when more than 30% of weights are pruned, and similarly when moving from the 30B to the 13B dense model
• IC-1387In-context learning capabilities in OPT and LLaMA models remain within 5% of dense-model accuracy even at 60-70% sparsity, and show less than 2% difference between the 30B and 1.3B dense OPT models
• IC-1388In LLaMA-13B, feed-forward layers are more critical than attention layers for fact recall, while both are equally important for in-context learning
• IC-1393Most mainstream LLMs generate value-violating content at high rates (APV 65-80%) across 2,397 morally ambiguous prompts, indicating substantial ethical misalignment
• IC-1428OPT 6.7B exhibits over 90% activation sparsity in FFN layers, reducing inference from 6.6G to 4.5G flops per token, while Llama 7B (SiLU) and Falcon 7B (GELU) show near-zero sparsity
• IC-1456Pre-trained language models fail to predict both interpretations of ambiguous inputs in zero-shot semantic parsing
• IC-1457Pre-trained language models track the distribution of logical forms in mixed few-shot prompts with conflicting examples
• IC-1496A single frozen transformer block from LLaMA-7B consistently improves performance across diverse visual tasks when appended to existing visual encoders
• IC-1497LLaMA-7B's frozen transformer block amplifies informative visual tokens, producing feature activations with higher Miou against ground-truth segmentation masks than both the baseline ViT and the model's own attention scores
• IC-1499Self-rationalization quality and task accuracy scale with model size across GPT-3, FLAN-T5, and LLaMA on five QA datasets
• IC-1508LLMs with in-context learning translate Kalamang-English at 44.7/45.8 CHRF, falling short of the human baseline of 51.6/57.0 CHRF
• IC-1509Kalamang-English translation performance on MTOb increases with model size within the Llama and Llama 2 families, and GPT-4 outperforms Text-davinci-003
• IC-1510Without retrieved context, LLMs are unable to translate Kalamang, and among context types, retrieved parallel sentences are most beneficial, followed by word list entries, then grammar book passages
• IC-1553GPT-J, GPT-2-XL, and Llama-13B decode approximately 48% of tested relations via a linear transformation on the subject representation, and this structure causally influences predictions
• IC-1584LLaMA-7B and GPT-J-6B fail to interpret textual emphasis markers, with marked prompting degrading performance substantially
• IC-1585LLaMA-7B and GPT-J-6B exhibit positional bias in instruction following: zero-shot performance varies significantly when the instruction is moved from after to before the context
• IC-1586In LLaMA-7B, steering all attention heads degrades JSON format accuracy below zero-shot, while steering a subset of 50-100 heads selected via multi-task profiling raises it to 96.64; performance varies dramatically across the 32 layers and individual heads
• IC-1630Llama and Pythia models represent entity-attribute bindings via additive binding id vectors that form a continuous subspace with metric structure
• IC-1631Binding id mechanism fidelity increases with model size in both Llama and Pythia families
• IC-626Llama 7B retains 95% of its common sense reasoning performance when compressed to 2 GB using JLCM
• IC-663OPT-1.3B, LLaMA-7B, and Aquila-7B encode sparse Harsanyi interactions, with only 29-51 salient interactions out of 1024 possible on SQuAD sentences
• IC-683LLaMA-7B, LLaMA-30B, Vicuna-7B, and Vicuna-13B achieve low accuracy (12.11% to 42.24%) in zero-shot and few-shot log-likelihood response evaluation
• IC-721The 72-head entity tracking circuit identified in Llama-7b achieves high faithfulness in Vicuna-7b and Goat-7b without any modification to the circuit graph
• IC-722Entity tracking in Llama-7b is implemented by detecting and transmitting the positional information of the correct entity, with distinct head groups for position detection, transmission, and value fetching
• IC-723The entity tracking performance gap between Goat-7b and Llama-7b is primarily attributable to enhanced positional information in the value fetcher and position transmitter heads
• IC-733CoT prompting improves factual accuracy for instruction-tuned LLMs but degrades it for non-instruction-tuned LLMs such as OPT, BLOOM, and LLaMA
• IC-747SOTA pruning methods (SparseGPT, Wanda, magnitude) cause significant degradation on knowledge-intensive tasks for Vicuna and Llama models at 25-30%+ unstructured sparsity, and fail completely for n:m structured sparsity
• IC-811Alpaca's 52k instruction-tuning data is predominantly low-quality (only 17.75% score ≥ 4.5 on accuracy), yet the full 52k data still yields higher MMLU scores than the filtered 9k subset for both 7b and 13b variants [builds-on]
• IC-815RLHF on general-purpose preference data increases stereotypical bias and decreases truthfulness in Pythia and Llama-7B models
• IC-816RLHF on general-purpose preference data increases privacy leakage in Pythia and Llama-7B models
• IC-817RLHF on general-purpose preference data improves machine ethics performance in Pythia and Llama-7B models
• IC-818RLHF on general-purpose preference data has negligible net effect on toxicity in Pythia and Llama-7B models
• IC-866A prefix applied to Llama-7B's first attention layer preserves the relative attention distribution over content positions and only adds a constant-direction bias to the attention block output
• IC-959LLaMA and OPT-1.3B (and Aquila-7B) encode more similar interaction primitives than smaller models such as BERT-base and BERT-large

Shared mechanisms

• Depth-dependent structurealso in Baichuan 2, BERT, BLIP-2, BLOOM, Chameleon, CLIP / CLIP-ViT (LC), DeepFloyd IF, DeiT-III, DINO, DINOv2, Falcon, Gemma, Gemma 2, GPT-2, GPT-J, GPT-NeoX-20B, Griffin, I3D, Idefics, InstructBLIP, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Phi, MAE, MAE-B/16, Mamba, MiniGPT-4, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, mPLUG-Owl, MPT, MultiBERTs, MViT V2, OLMo / OLMo base, OpenCLIP, OPT, Phi-2, Pythia, Qwen 2, Qwen2-VL, Qwen2.5, RoBERTa / RoBERTa-L, RWKV, SALMONN, SAM, SlowFast, Stable Diffusion, Swin Transformer, TerraMind, TimesFormer, TSM, Uniformer, Vicuna, VideoMAE, ViT, X3D, Yi
• Distance preservationalso in CLIP / CLIP-ViT (LC), CoPlace, DINO, DINOv2, Gemma, Gemma 2, ImageBind, LanguageBind, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, MAE, MAE-B/16, OpenCLIP, Pythia, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SigLIP, SLIP, TerraMind, ViT
• Failure modealso in AASIST, ADM, Aegis-Guard-Defensive, Alpaca, AnyLoc, AutoTikZ / DataTikZ, Baichuan, Baichuan 2, Baichuan2-13B, BakLLaVA, Bard, BEiT, BERT, BingChat, BLIP, BLIP-2, BLOOM, BSRNet, CF2, Chat-UniVi-7B, ChatGLM-6B / ChatGLM-6b-2, ChatGLM2, ChatGPT, CLAP, Claude 1.3, Claude 2.0, Claude 2.1, Claude 3, Claude 3.5, CLEAR, CLIP / CLIP-ViT (LC), CLIP4Clip, CLIPBERT, CLIPCap, CLMBR-T-BASE, CloFNet, Code Llama, CodeGeex2, CodeGen, CodeLlama-13B, CodeLlama-34B, CogVLM2, Cohere Command R, CoMEt, Command R+, CONCH, CycleGAN, DALL-E, DALL·E 2, DALL·E 3, DASR, DECAF, DeepSeek-2-Chat, DeepSeek-2-Coder, DeepSeek-V2-0628, DeepSeek-VL, DeepSeek-VL2, DeiT, DeiT-III, Depth Anything, DETR-R50, DimeNet++, DINO, DINO-FocalNet-Large, DINOv2, EGNN, Emu2, EquiformerV2, ESCN, ESM-2, ESM3, ESRGAN, EVA-CLIP, EVE, Falcon, Faster R-CNN / Faster R-CNN R50 / Faster R-CNN X101, FLAN-T5, Florence-2, FLUX / FLUX1, FoveaBox, Fuyu, Galactica-6.7B, GAT, 3D Gaussian Splatting, GCN, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GIN, GLIDE, GLM-4, GLM-4V, GloVe, GP-UNIT, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4.1, GPT-4o, GPT-J, GPT-NeoX-20B, GraphSAGE, Grounding DINO, Guanaco, GVP, Hawkeye, HiFaceGAN, HPS, HuggingGPT, IDDPM, Idefics, Idefics2, ImageBind, ImageBind-LLM-7B, Imagen Video, ImageReward, 12-in-1, InstructBLIP, InstructGPT, InternLM-2.5-7B, InternLM-XComposer2-VL, InternVideo, InternVL-1.5, InternVL2, Koala, LegalBERT, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, Llama-3-2-Vision, LLaMA-Adapter v2, Llama Guard, Llama Guard 2, Llama-Guard 3, Llama-VID, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, LLaVA-OneVision, LongVA-7B, LOVT, LWM-1M-JAX, MACE, MAE, Med-Flamingo, Merlot Reserve, MGCA, MiDaS, MiniCPM-V, MiniGPT-4, Mip-Splatting, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mistral Large 2, Mistral Large V2, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, MobileNetV2, Molmo, MolmoE-7B, Momentor, Moondream2, Moonshot-v1-8k, mPLUG-2, mPLUG-Owl, mPLUG-Owl3, mPLUG-Owl2, MPT, MSA Transformer, MultiBERTs, Nova Canvas, Nova Lite, Nova Pro, O1 / OpenAI-o1-preview, O3, O4-mini, OLMo / OLMo base, OneLLM, OpenAI Moderation, OpenChat-3.5-0106, OpenCLIP, OpenFlamingo, OPT, Otter, Otter-7B, PaLM 2, PaLM 62B, PandaGPT-7B, PerSAM, Phi-3, Phi-3.5 Mini Instruct, PickScore, PLIP, Prismatic, ProGen-2, Pythia, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen-VL, Qwen2.5, Qwen2-Audio, R2D2, RadFM, RCExplainer, RD-SR, RealESRNet, Reprover, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, RetinaNet-R50, RivaGAN, RS-LDS, RT-1, RT-2, SALMONN, SAM, SAM 2, SAULLM 54B, Scaffold-GS, SchNet, Seed-LLaMA-8B, SGC, SIREN, Sketch Transformer, SLD-max, SLD-medium, SLD-strong, SLDS, SLIMG, SpeechGPT, SphereNet, SRResNet, Stable Diffusion, StackLLaMA, Starcoder, StegaStamp, StyleGAN2-ADA, Swin Transformer, T5, TD-MPC, TerraMind, TimeChat, TranceptionEVE, TreeRing, Tulu 2, UnifiedQA, UniPerceiver, UNITER, UniVL, Van Gogh Diffusion, VERA, VGG / VGG13, Vicuna, Video-Chat-7B, Video-ChatGPT, Video-LLaMA, Video-LLaMA-2-13B, Video-LLaVA, VideoCLIP, ViLA-8B, ViLBERT, VindLU, VioLET, ViRTex, ViT, ViV1T, VTG-LLM, WildGuard, Wizardlm, X-CLIP, X-InstructBLIP-7B, XGen-MM, Xlm-R, Zephyr-7B-beta
• Feature interactionalso in Aquila-7B, CLIP / CLIP-ViT (LC), GPT-2, OPT, Sybil
• Linear representationalso in BLOOM, Cambrian-1, Chameleon, CLIP / CLIP-ViT (LC), DINOv2, EVA-CLIP, Falcon, Gemma, Gemma 2, GPT-2, GPT-J, HPSv2, ImageBind, InstructBLIP, LanguageBind, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, MAE, Mamba, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, OLMo / OLMo base, OpenCLIP, Phi-3, PickScore, Pythia, Qwen2-VL, Qwen2.5, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SALMONN, SAM, SigLIP, TerraMind, Tulu 2, Vicuna, ViT
• Positional biasalso in BERT, ChatGPT, Claude 3, Claude 3.5, Falcon, Fuyu, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GLIDE, GPT-2, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4.1, GPT-4o, GPT-J, InstructGPT, Llama 2 / Llama 2 base, Llama 3, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, MPT, O1 / OpenAI-o1-preview, O3, O4-mini, PaLM 2, Phi-3, Pythia, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Stable Diffusion, Sybil, Vicuna
• Scale-dependent behaviouralso in Aquila-7B, BEiT, BERT, BLOOM, Claude 2.1, Claude 3, Claude 3.5, CLIP / CLIP-ViT (LC), Code Llama, CodeGen, Cohere Command R, DeepSeek LLM, DeepSeekMoE, DeiT-III, DINO, DINOv2, EquiformerV2, ESCN, Falcon, FLAN-T5, Gemini 1.0 Pro, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, GPT-Neo, I3D, Idefics, InternLM-2.5-7B, InternLM-XComposer2-VL, InternLM2, InternVL-1.5, InternVL2, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, LongVA-7B, MAE, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Moirai, MPT, MViT V2, O1 / OpenAI-o1-preview, OLMo / OLMo base, OpenCLIP, OpenFlamingo, OpenLLaMA, OPT, PaLM 2, Phi-3, Platypus2-Instruct-70B, Pythia, Qwen, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen2.5, RedPajama-INCITE, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SlowFast, Solar 10.7B, Stable Diffusion, StableLM, Swin Transformer, T5, TerraMind, text-ada-001, TigerBot-70B-Chat, TimesFormer, TSM, Tulu 2, Uniformer, Vicuna, VideoMAE, ViLA-8B, Wizardlm, X3D, XGLM, Xwin-LM-70B-v0.1, Yi
• Shortcutalso in BakLLaVA, BLIP-2, Claude 3, Claude 3.5, CLIP / CLIP-ViT (LC), CLIP4Clip, CLIPBERT, DALL·E 2, DALL·E 3, DeepSeek-VL2, Eurus-RM-7B, Falcon, FLUX / FLUX1, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, iFlytekSpark-13B, InstructBLIP, Internlm2-Reward, InternVideo, InternVL2, Llama 2 / Llama 2 base, Llama 3, Llama-3-2-Vision, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, Med-Flamingo, Merlot Reserve, MiniGPT-4, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Molmo, mPLUG-2, mPLUG-Owl3, Nova Canvas, O1 / OpenAI-o1-preview, OpenCLIP, OPT, Otter, PaLM 62B, Pythia, Qwen, Qwen 2, Qwen-VL, Qwen2.5, RadFM, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, Stable Diffusion, Swin Transformer, Sybil, TerraMind, Tulu 2, UniPerceiver, UniVL, Vicuna, Video-LLaMA, VideoCLIP, VindLU, VioLET, X-CLIP