The paper applies DOCS to the expert weight matrices (W1, W2, W3) of the mixture-of-experts layers in Mixtral-8x7B. In many layers, typically just one expert stands out with clear separation (dark grids) from the other experts. The third expert in particular shows prominent dark intersecting lines in the similarity heatmaps, indicating it has a significantly different weight structure. The authors hypothesize this may relate to data load imbalance during MoE training.
Evidence
observational
Key metric
typically just one expert stands out, indicated by dark grids that show clear separation from the others; the third expert's prominent dark intersecting lines
Caveat
The authors note this is a hypothesis (data load imbalance) rather than a confirmed cause; the finding is based on visual inspection of heatmaps without a quantitative threshold for 'standing out'.