Logistic regression / Linear decoder (logistic regression) / Logistic classification / Logistic regression head

anchor

Linear classifier over the log-odds of the outcome. Used as a supervised baseline against which unsupervised or arithmetic constructions of a concept direction are compared.

Findings

• IC-1261LLaMA-2 attention to constraint tokens correlates with factual correctness, and a linear probe on these attention weights predicts factual errors comparably to model confidence [primary]
• IC-1392Title reproduction shows no contamination signal while tag reproduction shows a negative association with GitHub presence and a moderating difficulty effect for GPT-4 and GPT-3.5-turbo [primary]
• IC-332Llama-3-70B exhibits a friendlier, funnier, and less ethics-focused style than GPT-4 and Claude-3-Opus on Chatbot Arena, and these vibes predict model identity at 80% and user preference at 59% accuracy [eval]
• IC-413Factors of variation in ImageNet-X are linearly decodable from the second-to-last-layer representations of ImageNet-pretrained ResNet50 and ViT-B/16 [primary]
• IC-796GPT-2 Small MLP weight matrices are full-rank across all 12 layers and residual stream features are linearly recoverable from post-GELU MLP hidden activations, providing the structural conditions for the subspace patching illusion [primary]
• TM-002The mean-difference concept vector scores higher than trained classifiers under cosine similarity [compared-to]