anchor
Findings
- IC-054Gemma 2 2B performance degrades substantially when routed through Gemma Scope SAEs, and SAE-based feature suppression causes broad cross-domain degradation rather than targeted knowledge removal [eval]
- IC-181Truthfulness in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is linearly decodable from internal representations at exact answer tokens, with middle-to-late layers being most informative [eval]
- IC-182Truthfulness encoding in Mistral-7B, Mistral-7B-Instruct, Llama3-8B, and Llama3-8B-Instruct is skill-specific rather than universal; probing classifiers do not meaningfully generalize across different task types beyond logit-based baselines [eval]
- IC-36570B LLM variants tolerate substantially higher activation sparsity than smaller counterparts, and Llama-3 shows more degradation than Llama-2 and Mistral at 50% sparsity [eval]
- IC-370Knowledge entropy (sparsity of FFN memory coefficients) decreases consistently during pretraining for OLMo 1B, 7B, and Pythia 1.4B, and this decrease strongly correlates with reduced knowledge acquisition and increased forgetting in continual learning [eval]
- IC-371Artificially resuscitating inactive memory vectors by scaling the up-projection matrix K improves knowledge acquisition and reduces forgetting, with the effect more pronounced for later-stage OLMo models [eval]
- IC-401Ablating safety attention heads minimally degrades helpfulness on zero-shot tasks and also impairs course-correction capability in Llama-2-7b-chat [eval]
- IC-451Transformer block coupling of Jacobian singular vectors positively correlates with benchmark performance across 30+ LLMs, more strongly than parameter count, depth, or embedding dimension [eval]
- IC-452Transformer block coupling is absent at initialization and increases persistently throughout training in Pythia 12B and 6.9B, with layer-wise locality emerging [eval]
- IC-453Hidden representation trajectories in trained LLMs exhibit considerable linearity (mean LSS 4.25) compared to 6.54 at initialization, and linearity increases with training [eval]
- IC-472The degree of emergence metric derived from Pythia's internal structure positively correlates with benchmark performance across training epochs [eval]
- IC-626Llama 7B retains 95% of its common sense reasoning performance when compressed to 2 GB using JLCM [eval]