OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

Indraneil Paul, Falko Helm, Goran Glavaš, Iryna Gurevych|August 5, 2026arXiv

Key Takeaway

Training language models on code contexts that span multiple related files—rather than isolated snippets—dramatically improves their ability to handle long contexts and understand complex codebases, even when this data makes up a small fraction of total training.

Summary

OctoLong is a pipeline that creates long, dependency-rich code contexts by automatically retrieving related code files across repositories.

training data

Key Terms

mid-training context-extension repository-level-code-generation ast-parser instruction-tuning