Key-value cache precision and structure are the primary drivers of output divergence in language models, not the full decoder state—meaning you can replay intermediate reasoning stages reliably by preserving and transplanting caches across different precision levels.
This paper investigates why AI model outputs diverge when using cached intermediate states versus recomputing them from scratch. By systematically swapping key-value caches between different computation paths in a large language model, the authors show that cache precision and structure—not the full decoder state—primarily determines output differences.