IC-055OLMoE 6.9B experts show no domain specialization, with routing scores evenly distributed across MMLU domains, preventing targeted knowledge unlearning

Jungwoo Park, Ahn Young Jin, Kee-Eung Kim, Jaewoo Kang

SourceMonet: Mixture of Monosemantic Experts for Transformers

The paper examines OLMoE 6.9B's expert routing patterns on MMLU data. Using a skewness criterion (routing probability for a domain at least twice the second-highest), no experts were identified as specialized in any of the 14 MMLU domains. The authors note that OLMoE's expert routing scores are evenly distributed, reflecting feature polysemanticity due to its limited expert count. When they fall back to a maximum-activation criterion and remove the most-activated expert per domain, the resulting cross-domain performance changes (∆ others) range from -0.27 to -5.00, showing less targeted removal than the authors' MONET model.

Evidence

observational

Key metric

0 experts identified as domain-specialized under skewness criterion; ∆ others in MMLU ranges from -0.27 to -5.00 across 14 domains under max-activation criterion

Caveat

The skewness criterion (2× threshold) is the authors' own; a different threshold might yield different results. The paper acknowledges these criteria are 'basic and minimal.'

Model

OLMoE 6.9B

Datasets

MMLU / MMLU-Math [eval]

Related findings

IC-054

Extraction

automatic-extraction