Vladimir Fedosov, Aleksandr Sazhin, Artemiy Grinenko +1 more
Fine-tuning a language model on very long text runs into a wall: even with memory-saving tricks, standard attention gets expensive as the sequence grows, because every token attends to every other one. This paper stitches together a few ideas to stretch what fits on a modest GPU. They keep only the current segment differentiable in video memory, push older key-value data out to RAM or NVMe, and load back just a bounded set of exact historical tokens for each query block.
The concrete result: on a 16 GB card, plain training tops out around 2,048 tokens, while their method trains at 16,384 and can run inference through 131,072 tokens. Quality at the shared short length comes out essentially even with dense training, and throughput is already slightly better.
These figures are straight from the abstract on one specific setup, so read the paper before assuming they carry to yours.
Parameter-efficient fine-tuning reduces model and optimizer memory, but dense attention still makes long training sequences expensive. We combine Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered KV storage. Only the active segment remains differentiable in VRAM; older KV is detached into RAM or NVMe, and HGA loads a bounded set of exact historical tokens for each query block. On Qwen3-8B with 4-bit QLoRA and PG19, dense training on a 16 GB…
When Model Merging Rivals Joint Multi-Task Reinforcement Learning: A Task-Vector Geometry Analysis
arXiv (cs.CV) · July 17, 2026ArtChart: A Benchmark for Faithful Artistic Chart Generation with Integrated Text Rendering
arXiv (cs.AI) · July 17, 2026Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning
arXiv (cs.CV) · July 17, 2026Multi-Modal Semantic Segmentation of Electrolyzer Components for Sustainable Hydrogen Technologies: A Dual-Branch Deep Learning Approach