Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera +7 more
A model's tokenizer, the part that chops text into chunks, is frozen at the start of training and built around whatever languages mattered then. Add a language later and its words get shredded into many tiny pieces, which quietly makes things slower and more power-hungry for its speakers. Small on-device models feel this most, since their vocabulary tables eat into decoding bandwidth.
The authors grow a trained model's vocabulary in place, extending the original tokenizer's merge rules on multilingual text so old tokens carry over and each new one is built from old pieces, then doing a short two-stage retraining. On an 8B mixture-of-experts model, the result encodes Hindi and Vietnamese in about 2.4 and 2.6 times fewer tokens (up to 4 times for Thai), an estimated 2.2 to 3.7 times per-character speedup.
Those numbers come from the abstract, so check the paper for how they were measured.
A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary because the embedding and LM-head matrices are a small fraction of their parameters. On a…
Rendering 3D Gaussians on a Graph Processor
arXiv (cs.AI) · July 17, 2026When Not to Automate: A Formal Protocol for Human Preservation in AI-Optimized Organizations
arXiv (cs.LG) · July 17, 2026More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
arXiv (cs.CV) · July 17, 2026Handwritten and Printed Text Segmentation via Region-Aware Human-Writing Descriptor Engineering