When designing MoE models with many experts and shared components, you need to actively manage routing exploration—suppressing unnecessary experts per token can help the model find better expert combinations without getting lost in the combinatorial space.
This paper identifies a fundamental tension in modern Mixture-of-Experts models: fine-grained expert segmentation requires more exploration of the routing space, but shared experts reduce the variability needed for that exploration. The authors propose MirrorMoE, which uses token-adaptive expert suppression to balance these competing needs and improve performance across different model scales.