RoBERTa / RoBERTa-L

text · generative · anchor

Variants

RoBERTa-base, RoBERTa-Large

Findings

• IC-1005Social bias neurons in BERT-base-cased and RoBERTa-base are concentrated in the deepest transformer layers
• IC-1017Vision and language models pre-trained on noisy data exhibit degraded OOD transfer that is partially recoverable via SVD-based feature-space regularization
• IC-1409FF blocks in BERT and GPT-2 modify token-to-token contextualization, with the effect concentrated in specific layers and targeting specific linguistic compositions rather than simple word co-occurrence
• IC-1410FF's contextualization effects in BERT and GPT-2 are largely canceled by the residual connection and layer normalization, with LN's γ weights specifically shrinking the outlier dimensions in FF output
• IC-1544The latent spaces of pretrained foundational models across vision and text are not related by a single class of geometric transformations; the optimal alignment depends on the specific model pair, architecture, and dataset.
• IC-746RoBERTa-Large pretrained with different mask ratios exhibits a sweet spot in downstream accuracy on QNLI and SST-2

Shared mechanisms

• Depth-dependent structurealso in Baichuan 2, BERT, BLIP-2, BLOOM, Chameleon, CLIP / CLIP-ViT (LC), DeepFloyd IF, DeiT-III, DINO, DINOv2, Falcon, Gemma, Gemma 2, GPT-2, GPT-J, GPT-NeoX-20B, Griffin, I3D, Idefics, InstructBLIP, LLaMA, Llama 2 / Llama 2 base, Llama 3, Llama 3.1, Llama 3.2, Llama-3.2-3B, LLaVA, LLaVA-1.5 / LLaVA-v1.5, LLaVA-Phi, MAE, MAE-B/16, Mamba, MiniGPT-4, Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1, Mixtral 8x7B / Mistral 8x7B Instruct / Mixtral 46.7B / Mixtral 8x7B Instruct / Mixtral-instruct-8x7b, mPLUG-Owl, MPT, MultiBERTs, MViT V2, OLMo / OLMo base, OpenCLIP, OPT, Phi-2, Pythia, Qwen 2, Qwen2-VL, Qwen2.5, RWKV, SALMONN, SAM, SlowFast, Stable Diffusion, Swin Transformer, TerraMind, TimesFormer, TSM, Uniformer, Vicuna, VideoMAE, ViT, X3D, Yi