Low-Bit N-Gram Embeddings for Language Models

Stephen Chung, Wenyu Du|October 22, 2026arXiv

Key Takeaway

You can compress n-gram embedding tables to 4-bit precision and save 75% of their memory with minimal performance loss—or use the freed memory to make the model better overall.

Summary

This paper shows that n-gram embedding tables in language models can be stored at low precision (4 bits) instead of full precision, reducing memory usage by ~75% while keeping 99% of the performance benefit. The authors test quantization methods on models from scratch and on large existing models like DeepSeek, finding that saved memory can be reinvested to improve overall performance.

efficiency training architecture

Key Terms

quantization n-gram embedding-table low-precision-training