Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera +7 more
A model's tokenizer, the part that chops text into chunks, is frozen at the start of training and built around whatever languages mattered then. Add a language later and its words get shredded into many tiny pieces, which quietly makes things slower and more power-hungry for its speakers. Small on-device models feel this most, since their vocabulary tables eat into decoding bandwidth.
The authors grow a trained model's vocabulary in place, extending the original tokenizer's merge rules on multilingual text so old tokens carry over and each new one is built from old pieces, then doing a short two-stage retraining. On an 8B mixture-of-experts model, the result encodes Hindi and Vietnamese in about 2.4 and 2.6 times fewer tokens (up to 4 times for Thai), an estimated 2.2 to 3.7 times per-character speedup.
Those numbers come from the abstract, so check the paper for how they were measured.
A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary because the embedding and LM-head matrices are a small fraction of their parameters. On a…
CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach
arXiv (cs.LG) · July 17, 2026Constrained Hebbian Learning Supports Efficient Representational Allocation under Structural Constraints
arXiv (cs.AI) · July 17, 2026Candidate Attended Dialogue State Tracking Using BERT
arXiv (cs.LG) · July 17, 2026Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation
Plain-language explainers like this are written by Hevolve agents from the primary papers. Run agents like them locally — build by talking, keep your data on your own machine.