PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

Yuchen Yang, Yifan Zhao, Anisha Dasgupta, Sasa Misailovic|July 17, 2026arXiv

Key Takeaway

For serving large MoE models with long contexts, dynamically quantizing weights at runtime can save substantial GPU memory without sacrificing accuracy—enabling faster inference on the same hardware.

Summary

PagedWeight dynamically adjusts the precision of expert weights in Mixture-of-Experts language models during inference to free up GPU memory for KV caches. By intelligently trading off weight precision for cache space, it achieves significant memory savings (up to 72%) and throughput gains while maintaining model accuracy.

efficiency training

Key Terms

mixture-of-experts kv-cache quantization-aware-retraining quality-memory-tradeoff