Int8 quantization is a practical efficiency win for text embeddings with minimal quality loss, but binary quantization and aggressive dimensionality reduction hurt retrieval performance significantly—the trade-off varies by task type.
This study benchmarks how hierarchical text embeddings and quantization—two efficiency techniques—interact across 38 retrieval tasks. Testing 168 configurations per task, the authors find int8 quantization performs nearly as well as full precision (losing only 1.5% effectiveness), while binary quantization significantly hurts performance.