DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-AI, Aixin Liu, Bei Feng|May 7, 2024arXiv

Key Takeaway

MLA and fine-grained MoE routing enable frontier-class models at a fraction of the compute cost — DeepSeek-V2 pioneered the architecture that made DeepSeek-V3 and R1 possible.

Summary

Introduces Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, achieving strong performance while reducing inference costs dramatically. A 236B model with only 21B active parameters per token.

architecture efficiency scaling