You can fine-tune LLMs on 8x longer sequences with the same GPU by using hierarchical attention and tiered storage, without sacrificing model quality—the fine-tuned model performs identically to dense-attention training when evaluated.
This paper solves the problem of fine-tuning large language models on long sequences with limited GPU memory. By combining hierarchical attention with smart memory management—keeping only active training data in VRAM while storing older key-value pairs in RAM or NVMe—the method enables training on 16K tokens on a 16GB GPU where standard approaches fail at 4K tokens.