Probing classifiers / MLP probing classifiers / Q16 classifier

anchor

Freeze the model, train a small supervised model on its internal activations to predict a property of the input. The probe's capacity matters: a strong probe can recover information the model itself never uses, so linear and non-linear probes answer different questions. The anchor is Alain and Bengio, who introduced linear probes; Belinkov's 2021 survey (arXiv 2102.12452) is the other reference commonly cited for the same idea.

Findings

• IC-032Off-policy DPO causes a squeezing effect in LLMs where probability mass shifts to the most confident token, explaining degenerate repetition [eval]
• IC-043Five ~7B decoder-only LLMs develop a high-intrinsic-dimensionality phase in intermediate layers that marks the transition from surface-form to abstract linguistic processing, with earlier onset predicting better next-token prediction [primary]
• IC-1317Llama-2 and Pythia models contain linear representations of space and time that improve with depth and model scale [builds-on]
• IC-1391SLD concept removal variants and SD with negative prompts are bypassable by Ring-a-Bell adversarial prompts, increasing attack success rate from single digits to 90-100% for nudity [eval]
• IC-181Truthfulness in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is linearly decodable from internal representations at exact answer tokens, with middle-to-late layers being most informative [primary]
• IC-182Truthfulness encoding in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is skill-specific rather than universal; probing classifiers do not meaningfully generalize across different task types beyond logit-based baselines [primary]
• IC-183Error types in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct are linearly predictable from internal representations, encoding fine-grained information beyond binary correctness [primary]
• IC-184Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct can internally encode the correct answer while externally generating an incorrect one, with the discrepancy most pronounced for error types where the model shows no external preference for the correct answer [primary]
• TM-006Coordinates are recoverable from TerraMind's frozen features, latitude more accurately than longitude [primary]
• TM-007Coordinates are linearly decodable only in the larger TerraMind variants [primary]