You can train LLMs to handle longer contexts better using only self-generated data—no external corpora or teacher models needed—by having the model create its own reasoning tasks and learning from them.
LongZero enables LLMs to improve their long-context reasoning abilities without external data by having models generate their own training tasks. The method creates multi-hop reasoning and retrieval tasks by first synthesizing compact question-answer pairs with evidence, then expanding that evidence into long multi-document contexts, before training with reinforcement learning.