GPT-2

text · generative

Variants

GPT-2 Small, GPT-2 Large, GPT-2 Medium, GPT-2 XL, GPT-2 XL 1.5B

Findings

• IC-052GPT-2 small's IOI circuit activations are linearly decomposable into features for the io, s, and pos attributes, with the l10h0 name mover's attention decomposing into sparse pairwise feature interactions
• IC-053In GPT-2 small's l10h0 name mover queries, the io attribute is encoded with higher-magnitude features than the s attribute, and both are causally relevant, but SAEs preferentially learn io features due to the magnitude asymmetry
• IC-1015GPT-J and 10 other LLMs exhibit overthinking: calibrated accuracy given incorrect few-shot demonstrations peaks at a critical layer then declines, and ablating 5 false induction heads in late layers reduces the accuracy gap by 38.9% on average
• IC-1017Vision and language models pre-trained on noisy data exhibit degraded OOD transfer that is partially recoverable via SVD-based feature-space regularization
• IC-1127The LM head in GPT-2, GPT-J, BLOOM, Pythia, and LLaMA-2 projects all input token hidden states into interpretable token distributions over the vocabulary, and these distributions converge approximately monotonically toward the final layer's distribution
• IC-1199GPT-2 next-token distributions contain correctable tail errors from the softmax bottleneck that degrade generation quality under low-entropy sampling, with basis-aware threshold sampling improving MAUVE across all four sizes
• IC-1200GPT-2-XL's untruncated next-token log-probability matrix has rank saturating at its hidden dimensionality of 1600, while truncation sampling produces post-truncation distributions whose estimated rank grows far beyond 1600
• IC-1206GPT-2-XL, GPT-J, Falcon-7B, Llama-2-7B, and Llama-2-13B are vulnerable to backdoor injection via lightweight parameter editing with only 15 samples, achieving near-100% attack success rate while preserving clean performance
• IC-1207For GPT-2-XL, backdoor injection via parameter editing is most effective on intermediate layers (15-35) and notably less effective on the first 10 and last 5 layers
• IC-1208Larger LLMs (Llama-2-13B) require more data samples for successful backdoor injection via parameter editing compared to smaller models (GPT-2-XL 1.5B)
• IC-1219Frozen GPT-2 XL contains pre-trained attention heads that implement the nearest-neighbor algorithm
• IC-1221LLM performance on in-context boolean function learning is scale-dependent, with GPT-2 failing and Llama-2 models improving gradually with size
• IC-1231On APPS, released code generation models span pass@1 from 0.20 (GPT-3 175B) to 6.20 (CodeRL), with value-based and policy-based RL methods outperforming supervised baselines
• IC-1232GPT-2 XL and GPT-J exhibit knowledge conflict when subjected to reverse and composite knowledge edits, with ROME and MEMIT showing near-total failure on reverse edits
• IC-1233GPT-2 XL and GPT-J exhibit irreversible knowledge distortion after round-editing, with the effect being more severe when the edit target is semantically distant from the true labels
• IC-1250GPT-2-medium shares 78% of its top attention heads between the IOI circuit and the colored objects circuit
• IC-1251Intervening on four attention heads in GPT-2-medium boosts colored objects accuracy from 49.6% to 93.7% by making the circuit behave like the IOI circuit
• IC-1252Circuit overlap between IOI and colored objects in GPT-2 decreases as model scale increases from medium to xl
• IC-1320GPT-2 XL (1.5B) exhibits better calibration than larger models from the LLaMA, LLaMA2, and GPT-J families despite having fewer parameters
• IC-1369Successor heads that increment ordinal-sequence tokens exist in Pythia, GPT-2, and Llama-2 models from 31M to 12B parameters
• IC-1370MLP0 representations of ordinal-sequence tokens in Pythia-1.4b contain linearly decodable mod-10 features that are causally important for incrementation
• IC-1389Video-language models do not significantly outperform image-language models on temporal reasoning tasks in VILMA
• IC-1390Proficiency tests reveal that a substantial portion of correct main-test predictions by VidLMs and ILMs are spurious rather than reflecting robust understanding
• IC-1409FF blocks in BERT and GPT-2 modify token-to-token contextualization, with the effect concentrated in specific layers and targeting specific linguistic compositions rather than simple word co-occurrence
• IC-1410FF's contextualization effects in BERT and GPT-2 are largely canceled by the residual connection and layer normalization, with LN's γ weights specifically shrinking the outlier dimensions in FF output
• IC-1518Domain finetuning of LLaMA 2 7B, LLaMA 2 13B, and GPT-2 XL on PubMed causes topic and style priors to shift dramatically, accounting for the majority of the probability change, while factual knowledge learning contributes only a small fraction
• IC-1553GPT-J, GPT-2-XL, and Llama-13B decode approximately 48% of tested relations via a linear transformation on the subject representation, and this structure causally influences predictions
• IC-212GPT-2-small and CPM-distilled converge on nearly identical IOI circuits despite being trained independently on English and Chinese
• IC-247Pretrained LLMs are invariant to positional shifts but sensitive to duration scaling of the input
• IC-256GPT-2 small's attention product functions p_i^T k^T q p_j are approximately translation-invariant across all 144 heads
• IC-282GPT-2 XL (1.5B) exhibits lower accuracy but reduced overconfidence (smaller ECE and Brier scores) compared to larger models on the CAT benchmark
• IC-296The degree to which SAE features are active at multiple residual-stream layers increases with model size in Pythia, Gemma 2, Llama 3.2, and GPT-2
• IC-348Sequential parameter-modifying editing causes progressive degradation of general abilities in GPT-2 XL, Llama-2 7B, and Llama-3 8B, driven by growth in the condition number of the edited matrix
• IC-349Larger LLMs (Llama-2 7B, Llama-3 8B) suffer more severe general ability degradation than smaller models (GPT-2 XL 1.5B) under the same number of sequential edits
• IC-358GPT-2-small and Mistral 7B contain circular representations of days of the week and months of the year in their internal activations, discovered via SAE dictionary element clustering
• IC-360GPT-2 achieves only trivial accuracy on modular arithmetic tasks for days of the week and months of the year despite containing circular representations
• IC-372Language models universally decompose retrieval tasks into request processing in middle layers and entity retrieval in late layers at the last token position
• IC-425Pretrained GPT-2 Large fails at structural in-context learning on unseen tokens in a syllogism task
• IC-430Toxicity is linearly separable in the context embedding space of LLMs (Llama-2-7b, GPT-2-large, Llama-3.1-8B-Instruct), with the instruction-tuned model showing a stronger signal
• IC-451Transformer block coupling of Jacobian singular vectors positively correlates with benchmark performance across 30+ LLMs, more strongly than parameter count, depth, or embedding dimension
• IC-453Hidden representation trajectories in trained LLMs exhibit considerable linearity (mean LSS 4.25) compared to 6.54 at initialization, and linearity increases with training
• IC-454Most hidden trajectories in trained LLMs exhibit exponential growth in norm as a function of depth, a property that emerges with training
• IC-462GPT-2 encodes toxicity in a low-dimensional linear subspace of its MLP layers, concentrated in higher layers
• IC-463DPO's first-step gradients in GPT-2 are correlated with the toxic subspace, with stronger alignment in later layers and with more samples
• IC-525GPT-2 small's residual stream at layer 8 decomposes into two sub-spaces of approximately 25% and 75% of the dimensionality
• IC-526GPT-2 small's first token position has residual stream norms more than an order of magnitude larger than all other positions
• IC-528The knowledge localization assumption fails for a large fraction of facts in GPT-2, Llama2-7B, and Llama3-8B, with 77% of facts classified as inconsistent knowledge in Llama3-8B
• IC-529For inconsistent knowledge in GPT-2, Llama2-7B, and Llama3-8B, the knowledge neurons are associated with the specific query rather than the fact, as shown by differential effects of suppressing or enhancing query-specific versus neighbor neurons
• IC-530The attention module in GPT-2, Llama2-7B, and Llama3-8B plays a selective role in knowledge expression by activating specific knowledge neurons for a given query, as demonstrated by suppressing or enhancing attention scores at knowledge synapse positions
• IC-608Token trajectories in GPT-2, Llama 2 7B, Mistral 7B, and Llama 3.2 models cluster on a low-dimensional manifold and follow a linear drift plus Gaussian noise dynamics
• IC-636Syntactic phenomena (determiner-noun and subject-verb agreement) localize to the same topmost-layer MLP neurons as factual information in BERT, GPT-2, and Llama-2
• IC-638ROME editing on GPT-2 XL and Llama-2 7B achieves high reliability but fails under bijective symmetry (23.71%–33.64%) and synonymous invariance (52.35%–58.36%) criteria
• IC-639The causal tracing pattern of MLP at early layers and attention at late layers is not stable across factual and syntactic phenomena in GPT-2 XL
• IC-715Factual information deleted from GPT-J, LLaMA-2, and GPT-2-XL via ROME or MEMIT remains linearly recoverable from intermediate hidden states, with up to 89% extraction success at budget b=20
• IC-716Factual information deleted from GPT-J, LLaMA-2, and GPT-2-XL via ROME or MEMIT is recoverable by sampling outputs on automatically generated rephrased prompts, with up to 56% extraction success at budget b=20
• IC-7951D subspaces of MLP activations found by DAS in GPT-2 Small (IOI) and GPT-2 XL (factual recall) produce apparent causal effects that are interpretability illusions driven by causally disconnected components activating dormant pathways
• IC-796GPT-2 Small MLP weight matrices are full-rank across all 12 layers and residual stream features are linearly recoverable from post-GELU MLP hidden activations, providing the structural conditions for the subspace patching illusion
• IC-840GPT-2 Small's name mover heads exhibit disrupted attention patterns under out-of-distribution Gaussian noise corruption
• IC-849GPT-2 and T5-base exhibit vanishing expected gradients under RFT for inputs with small reward standard deviation, prevalent in 3 of 7 GRUE datasets, causing RFT to underperform SFT
• IC-850A partial SFT phase (40% of steps, 1% of samples) before RFT allows GPT-2 and T5-base to reach 96% of the reward achieved with full SFT+RFT, by reducing the number of inputs with vanishing gradients
• IC-867In GPT-2 prefix-tuned on the emotion dataset, attention over prefix positions is nearly constant across inputs, collapsing the effective bias subspace to a single direction in most layers
• IC-903Knowledge editing performance (ES, GS, LS) improves as model scale increases from GPT-2 (124M) to T5-XL (2.8B) to GPT-J (6B) across all editing methods
• IC-936GPT-2's learned positional encodings cause context vectors to lose linear independence after one layer, whereas BERT's sinusoidal encodings preserve it

Shared mechanisms

• Circular representationalso in Gemini 1.0 Pro, Gemma 2, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, Llama 3, Llama 3.1, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, TerraMind
• Depth-dependent structurealso in Baichuan 2, BERT, BLIP-2, BLOOM, Chameleon, CLIP / CLIP-ViT (LC), DeepFloyd IF, DeiT-III, DINO, DINOv2, Falcon, Gemma, Gemma 2, GPT-J, GPT-NeoX-20B, Griffin, I3D, Idefics, InstructBLIP, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Phi, MAE, MAE-B/16, Mamba, MiniGPT-4, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, mPLUG-Owl, MPT, MultiBERTs, MViT V2, OLMo / OLMo base, OpenCLIP, OPT, Phi-2, Pythia, Qwen 2, Qwen2-VL, Qwen2.5, RoBERTa / RoBERTa-L, RWKV, SALMONN, SAM, SlowFast, Stable Diffusion, Swin Transformer, TerraMind, TimesFormer, TSM, Uniformer, Vicuna, VideoMAE, ViT, X3D, Yi
• Failure modealso in AASIST, ADM, Aegis-Guard-Defensive, Alpaca, AnyLoc, AutoTikZ / DataTikZ, Baichuan, Baichuan 2, Baichuan2-13B, BakLLaVA, Bard, BEiT, BERT, BingChat, BLIP, BLIP-2, BLOOM, BSRNet, CF2, Chat-UniVi-7B, ChatGLM-6B / ChatGLM-6b-2, ChatGLM2, ChatGPT, CLAP, Claude 1.3, Claude 2.0, Claude 2.1, Claude 3, Claude 3.5, CLEAR, CLIP / CLIP-ViT (LC), CLIP4Clip, CLIPBERT, CLIPCap, CLMBR-T-BASE, CloFNet, Code Llama, CodeGeex2, CodeGen, CodeLlama-13B, CodeLlama-34B, CogVLM2, Cohere Command R, CoMEt, Command R+, CONCH, CycleGAN, DALL-E, DALL·E 2, DALL·E 3, DASR, DECAF, DeepSeek-2-Chat, DeepSeek-2-Coder, DeepSeek-V2-0628, DeepSeek-VL, DeepSeek-VL2, DeiT, DeiT-III, Depth Anything, DETR-R50, DimeNet++, DINO, DINO-FocalNet-Large, DINOv2, EGNN, Emu2, EquiformerV2, ESCN, ESM-2, ESM3, ESRGAN, EVA-CLIP, EVE, Falcon, Faster R-CNN / Faster R-CNN R50 / Faster R-CNN X101, FLAN-T5, Florence-2, FLUX / FLUX1, FoveaBox, Fuyu, Galactica-6.7B, GAT, 3D Gaussian Splatting, GCN, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GIN, GLIDE, GLM-4, GLM-4V, GloVe, GP-UNIT, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4.1, GPT-4o, GPT-J, GPT-NeoX-20B, GraphSAGE, Grounding DINO, Guanaco, GVP, Hawkeye, HiFaceGAN, HPS, HuggingGPT, IDDPM, Idefics, Idefics2, ImageBind, ImageBind-LLM-7B, Imagen Video, ImageReward, 12-in-1, InstructBLIP, InstructGPT, InternLM-2.5-7B, InternLM-XComposer2-VL, InternVideo, InternVL-1.5, InternVL2, Koala, LegalBERT, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, Llama-3-2-Vision, LLaMA-Adapter v2, Llama Guard, Llama Guard 2, Llama-Guard 3, Llama-VID, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, LLaVA-OneVision, LongVA-7B, LOVT, LWM-1M-JAX, MACE, MAE, Med-Flamingo, Merlot Reserve, MGCA, MiDaS, MiniCPM-V, MiniGPT-4, Mip-Splatting, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mistral Large 2, Mistral Large V2, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, MobileNetV2, Molmo, MolmoE-7B, Momentor, Moondream2, Moonshot-v1-8k, mPLUG-2, mPLUG-Owl, mPLUG-Owl3, mPLUG-Owl2, MPT, MSA Transformer, MultiBERTs, Nova Canvas, Nova Lite, Nova Pro, O1 / OpenAI-o1-preview, O3, O4-mini, OLMo / OLMo base, OneLLM, OpenAI Moderation, OpenChat-3.5-0106, OpenCLIP, OpenFlamingo, OPT, Otter, Otter-7B, PaLM 2, PaLM 62B, PandaGPT-7B, PerSAM, Phi-3, Phi-3.5 Mini Instruct, PickScore, PLIP, Prismatic, ProGen-2, Pythia, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen-VL, Qwen2.5, Qwen2-Audio, R2D2, RadFM, RCExplainer, RD-SR, RealESRNet, Reprover, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, RetinaNet-R50, RivaGAN, RS-LDS, RT-1, RT-2, SALMONN, SAM, SAM 2, SAULLM 54B, Scaffold-GS, SchNet, Seed-LLaMA-8B, SGC, SIREN, Sketch Transformer, SLD-max, SLD-medium, SLD-strong, SLDS, SLIMG, SpeechGPT, SphereNet, SRResNet, Stable Diffusion, StackLLaMA, Starcoder, StegaStamp, StyleGAN2-ADA, Swin Transformer, T5, TD-MPC, TerraMind, TimeChat, TranceptionEVE, TreeRing, Tulu 2, UnifiedQA, UniPerceiver, UNITER, UniVL, Van Gogh Diffusion, VERA, VGG / VGG13, Vicuna, Video-Chat-7B, Video-ChatGPT, Video-LLaMA, Video-LLaMA-2-13B, Video-LLaVA, VideoCLIP, ViLA-8B, ViLBERT, VindLU, VioLET, ViRTex, ViT, ViV1T, VTG-LLM, WildGuard, Wizardlm, X-CLIP, X-InstructBLIP-7B, XGen-MM, Xlm-R, Zephyr-7B-beta
• Feature interactionalso in Aquila-7B, CLIP / CLIP-ViT (LC), LLaMA, OPT, Sybil
• Linear representationalso in BLOOM, Cambrian-1, Chameleon, CLIP / CLIP-ViT (LC), DINOv2, EVA-CLIP, Falcon, Gemma, Gemma 2, GPT-J, HPSv2, ImageBind, InstructBLIP, LanguageBind, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, MAE, Mamba, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, OLMo / OLMo base, OpenCLIP, Phi-3, PickScore, Pythia, Qwen2-VL, Qwen2.5, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SALMONN, SAM, SigLIP, TerraMind, Tulu 2, Vicuna, ViT
• Method artefactalso in Baichuan, CLIP / CLIP-ViT (LC), ConvNeXt, EfficientNet, Falcon, Gemma, Gemma 2, GPT-3.5 / ChatGPT-3.5, GPT-4o, GPT-J, InternLM, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, MAP-NEO, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, OLMo / OLMo base, OpenCLIP, OpenLLaMA, OPT, Pythia, Qwen1.5, Qwen 2, Qwen2.5, RedPajama, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, Skywork, Stable Diffusion, StableLM, TerraMind, ViT, Yi, ZiYA2
• Positional biasalso in BERT, ChatGPT, Claude 3, Claude 3.5, Falcon, Fuyu, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GLIDE, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4.1, GPT-4o, GPT-J, InstructGPT, LLaMA, Llama 2 / Llama 2 base, Llama 3, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, MPT, O1 / OpenAI-o1-preview, O3, O4-mini, PaLM 2, Phi-3, Pythia, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Stable Diffusion, Sybil, Vicuna
• Register tokensalso in DeiT-III, DINO, DINOv2, MAE, OpenCLIP, TerraMind
• Scale-dependent behaviouralso in Aquila-7B, BEiT, BERT, BLOOM, Claude 2.1, Claude 3, Claude 3.5, CLIP / CLIP-ViT (LC), Code Llama, CodeGen, Cohere Command R, DeepSeek LLM, DeepSeekMoE, DeiT-III, DINO, DINOv2, EquiformerV2, ESCN, Falcon, FLAN-T5, Gemini 1.0 Pro, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, GPT-Neo, I3D, Idefics, InternLM-2.5-7B, InternLM-XComposer2-VL, InternLM2, InternVL-1.5, InternVL2, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA-1.5 / LLaVA-v1.5, LLaVA-NeXT / LLaVA 1.6, LongVA-7B, MAE, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Moirai, MPT, MViT V2, O1 / OpenAI-o1-preview, OLMo / OLMo base, OpenCLIP, OpenFlamingo, OpenLLaMA, OPT, PaLM 2, Phi-3, Platypus2-Instruct-70B, Pythia, Qwen, Qwen1.5, Qwen 2, Qwen 2.5 72B Instruct, Qwen2-VL, Qwen-Audio, Qwen2.5, RedPajama-INCITE, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, SlowFast, Solar 10.7B, Stable Diffusion, StableLM, Swin Transformer, T5, TerraMind, text-ada-001, TigerBot-70B-Chat, TimesFormer, TSM, Tulu 2, Uniformer, Vicuna, VideoMAE, ViLA-8B, Wizardlm, X3D, XGLM, Xwin-LM-70B-v0.1, Yi
• Shortcutalso in BakLLaVA, BLIP-2, Claude 3, Claude 3.5, CLIP / CLIP-ViT (LC), CLIP4Clip, CLIPBERT, DALL·E 2, DALL·E 3, DeepSeek-VL2, Eurus-RM-7B, Falcon, FLUX / FLUX1, Gemini, Gemini 1.5 / Gemini Pro 1.5, Gemma, Gemma 2, GPT-3 / GPT base, GPT-3.5 / ChatGPT-3.5, GPT-4 / ChatGPT4 / GPT-4 Code Interpreter / GPT-4 Technical Report, GPT-4o, GPT-J, iFlytekSpark-13B, InstructBLIP, Internlm2-Reward, InternVideo, InternVL2, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama-3-2-Vision, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Med, LLaVA-NeXT / LLaVA 1.6, Med-Flamingo, Merlot Reserve, MiniGPT-4, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, Molmo, mPLUG-2, mPLUG-Owl3, Nova Canvas, O1 / OpenAI-o1-preview, OpenCLIP, OPT, Otter, PaLM 62B, Pythia, Qwen, Qwen 2, Qwen-VL, Qwen2.5, RadFM, ResNet / ResNet-152 / ResNet-101 / ResNet-50-BN, Stable Diffusion, Swin Transformer, Sybil, TerraMind, Tulu 2, UniPerceiver, UniVL, Vicuna, Video-LLaMA, VideoCLIP, VindLU, VioLET, X-CLIP