RAG systems frequently misattribute claims to retrieved sources when they actually come from parametric memory, and current citation verification methods cannot reliably detect this—a critical issue for trustworthy AI systems.
This paper reveals that retrieval-augmented generation (RAG) systems often cite sources they didn't actually use, instead relying on knowledge baked into the model itself. By replacing facts in retrieved passages with false versions and re-querying the model, researchers found that 1.2–15.7% of cited claims come from the model's internal memory rather than the provided context.