Sparse MoE at practical scale works — Mixtral showed that routing tokens to a subset of experts gives large-model quality at small-model inference cost.
Introduces Mixtral 8x7B, a sparse MoE model that routes each token to 2 of 8 expert networks. Matches or outperforms Llama 2 70B and GPT-3.5 while using only 13B active parameters per token.