Memory in LLMs can hurt performance even when perfectly accurate and relevant—the problem isn't retrieval quality but how retrieved information biases reasoning. A simple prompt-based fix can recover performance without redesigning memory systems.
This paper identifies and benchmarks 'cognitive traps' where LLMs with memory systems retrieve accurate, relevant information that paradoxically harms reasoning on current tasks. The authors introduce MemTrapBench to evaluate two failure modes (reasoning fixation and belief distortion) and propose AdaptiveMem, an inference-time method to mitigate these traps while maintaining memory benefits.