To internalize document knowledge without retrieval, separate the training into three stages: inject documents as training data, align with QA supervision, then merge back with the base model to avoid catastrophic forgetting of general skills.
This paper tackles the problem of making LLMs answer questions about specific documents without retrieving them at inference time. The authors propose IAR, a three-stage training method that injects document knowledge, aligns the model to answer questions correctly, and recovers general abilities by merging with the base model.