You can dramatically compress KV caches (by 32×) while maintaining retrieval accuracy by treating the cache as a learnable component and fine-tuning it on the document itself—without knowing what questions will be asked.
This paper introduces Latent Compaction, a method that compresses key-value caches in large language models by learning a compact, document-specific cache through test-time training.