Using multiple routers with different expert preferences improves expert selection in MoE models, and reusing routing signals to customize expert inputs boosts performance without increasing compute cost.
This paper improves sparse mixture-of-experts models by using multiple routers to select experts and modulate their computation. Instead of one router deciding which experts to use, multiple independently-trained routers vote on expert selection, and their preferences are reused to customize each expert's input representation.