Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

William Fedus, Barret Zoph, Noam Shazeer|January 11, 2021arXiv

Key Takeaway

Sparse MoE with single-expert routing enables trillion-parameter models at practical compute costs — the architectural pattern behind modern MoE models like Mixtral.

Summary

Simplifies Mixture-of-Experts routing by using a single expert per token instead of top-k. This Switch routing enables scaling to trillion-parameter models while keeping computational cost manageable.

architecture scaling efficiency