You can quantize LLM weights and KV caches to 2-3 bits without accuracy loss by selectively streaming precision from slower storage during inference—the key is overlapping transfers with computation so decoding speed doesn't suffer.
StrataFlux recovers accuracy lost to extreme quantization (2-4 bits) in LLM inference by strategically fetching higher-precision data from RAM and SSDs during generation. It overlaps these transfers with computation so they don't slow down decoding, enabling models like Llama-3.1-8B to run on GPUs with half the memory while matching the accuracy of less-aggressive quantization schemes.