Linear Probing / Ridge regression linear probing / Linear probe / Linear probe fine-tuning / Linear regression probing / Linear ridge regression probes / Supervised probing / ERM linear probe

anchor

Findings

• IC-1017Vision and language models pre-trained on noisy data exhibit degraded OOD transfer that is partially recoverable via SVD-based feature-space regularization [compared-to]
• IC-1105PAC-Bayes generalization bounds for discrete class prompts on CLIP are within a few percentage points of the actual test error across CIFAR-10, CIFAR-100, ImageNet, FMOW, and OfficeHome [compared-to]
• IC-1106CLIP prompts found by greedy search do not fit random labels: train and test error drop in tandem as the fraction of flipped labels increases, unlike a linear probe which achieves near-random accuracy [compared-to]
• IC-122Concept representations in Llama-2-7B, Gemma-7B, and Llama-2-13B become more consistent in deeper layers [primary]
• IC-1226SAM's ViT-B encoder achieves 54.2% ImageNet-1k linear probing accuracy versus 67.7% for MAE's ViT-B, indicating its segmentation pretraining impairs high-level semantic representation [primary]
• IC-123Llama-2-7B, Gemma-7B, and Llama-2-13B organize 16 concepts into hierarchical clusters in their representation space that reflect real-world category structure [primary]
• IC-130Amplifying visual-token attention in high layers (16-32) of released VLMs reduces sycophancy while preserving VQA accuracy, indicating that insufficient high-layer visual attention is a key cause of sycophancy [supporting]
• IC-1317Llama-2 and Pythia models contain linear representations of space and time that improve with depth and model scale [primary]
• IC-1370MLP0 representations of ordinal-sequence tokens in Pythia-1.4b contain linearly decodable mod-10 features that are causally important for incrementation [validation]
• IC-161Linear probes on Pythia-70m and Gemma-2-2b trained on the ambiguous Bias in Bios set rely on gender as a spurious feature, with gender accuracy far exceeding profession accuracy [primary]
• IC-218LLaMA3-8B and other LLMs solve arithmetic via a bag of independent heuristic neurons in middle and late MLP layers rather than a robust algorithm [primary]
• IC-257327 DNNs approach or exceed human accuracy on object depth order but are near chance on VPT-basic, while humans show the opposite pattern [primary]
• IC-258DNN accuracy on 3D perception tasks correlates with ImageNet object classification accuracy, suggesting 3D cues emerge as a byproduct of object recognition training [primary]
• IC-493A linear direction in the input embedding space of Llama-2-7B-Chat, Llama-2-13B-Chat, Mistral-7B-Instruct-v0.3, and Phi-3-mini-128k predicts instruction-following success, generalizes across tasks but not instruction types, and can be used to improve adherence via representation engineering [primary]
• IC-501Linear probes on middle-layer attention heads of Llama-2-7B-Chat, Mistral-7B-Instruct-v0.1, and Vicuna-7B-v1.5 predict US lawmakers' DW-Nominate ideology scores with Spearman correlations around 0.85 [primary]
• IC-502Linear probes trained on US lawmaker ideology generalize to predict Ad Fontes media slant scores when the same models simulate news outlets [primary]
• IC-503Adding probe regression coefficients to attention head activations steers Llama-2-7B-Chat, Mistral-7B-Instruct-v0.1, and Vicuna-7B-v1.5 toward more liberal or conservative generated text [supporting]
• IC-646DINOv2, DeiT-III, and OpenCLIP repurpose approximately 2% of patch tokens in low-informative background areas as internal registers, discarding local patch information while aggregating global image information; DINO does not exhibit this behaviour [primary]
• IC-685CLIP ViT-B/32 with a linear probe relies on gender as a spurious correlation for hair color, achieving only 15.85% accuracy on female gray hair [primary]
• IC-737CLIP ViT-B/16 binarized dot products yield 0.50–0.58 accuracy on binary concept presence queries across five image classification datasets [validation]
• IC-746RoBERTa-Large pretrained with different mask ratios exhibits a sweet spot in downstream accuracy on QNLI and SST-2 [eval]
• IC-796GPT-2 Small MLP weight matrices are full-rank across all 12 layers and residual stream features are linearly recoverable from post-GELU MLP hidden activations, providing the structural conditions for the subspace patching illusion [primary]
• IC-834CLIP zero-shot predictions exhibit high equal opportunity difference when target and sensitive attributes are intrinsically dependent [compared-to]