For LLM semantic caches, focus on validating answer quality before optimizing eviction policy; LFU is a solid default, and geometry-aware policies don't help much because newly cached queries rarely have similar neighbors.
This paper evaluates different cache eviction policies for semantic caches that store LLM responses based on query embeddings. Testing seven policies across multiple datasets and cache sizes, the authors find that LFU (Least Frequently Used) performs best, but no policy significantly outperforms it.