Big multilingual models tend to stumble on languages written in non-Latin scripts, and much of the problem sits upstream in the tokenizer. Trained mostly on Latin text, it lacks proper pieces for a script like Ge'ez, so Amharic and Tigrinya words get chopped into far too many fragments and many characters fall outside its vocabulary.

VEXMLM patches this by teaching XLM-R the script directly. The authors build a tokenizer on Amharic and Tigrinya text, bolt 30,000 new Ge'ez subwords onto the vocabulary, and give each new piece a sensible starting embedding by averaging the fragments it used to be split into. After continued training and task tuning, question answering jumps from 66 to 87 exact-match on the two languages, and recognition of entities built from unseen tokens climbs from 81% to 94%. The gains also carry to 17 related African languages they never specifically trained on.

Those figures come from the abstract, so read the paper for the full evaluation.