You can compress LLM key-value caches to 4-bit precision without significant accuracy loss by carefully handling quantization errors at the algorithm level and co-designing with hardware capabilities—delivering major serving speedups in practice.
BitDecoding-FP4 enables efficient long-context LLM serving by storing key-value caches in ultra-low-precision 4-bit floating-point format while maintaining accuracy.