Modelpedia
Work in progress
About
Findings
Models
Concepts
Methods
Datasets
Sources
Light
Dark
Monet: Mixture of Monosemantic Experts for Transformers
2025-01-22
· ICLR 2025 Poster ·
anchor
Findings
IC-054
Gemma 2 2B performance degrades substantially when routed through Gemma Scope SAEs, and SAE-based feature suppression causes broad cross-domain degradation rather than targeted knowledge removal
IC-055
OLMoE 6.9B experts show no domain specialization, with routing scores evenly distributed across MMLU domains, preventing targeted knowledge unlearning