MoE models are significantly more vulnerable to data repetition than dense models—a critical concern as training data becomes scarce. Regularization helps, but the fundamental mismatch between sparsity and repeated data suggests new architectural approaches are needed.
This paper investigates how Mixture-of-Experts (MoE) models—which use sparse, specialized sub-networks—overfit more severely than dense models when training data is repeated.