Paul-Andrei Pogăcean, Sanda-Maria Avram
Figuring out which language a text is written in usually means either a heavy neural network or a lightweight model that counts how often letters and letter pairs appear. The counting approach is fast, but there is a catch: letter frequencies are proportions that sum to one, and treating them as ordinary points in space distorts the distances between them.
This work borrows a fix from compositional data analysis, running the frequencies through a centered log-ratio transformation into a space where plain Euclidean distance behaves like a proper distance for proportions. Add Laplace smoothing for rare combinations and you get a classifier that runs in linear time. Tested on six languages, it stays accurate across text lengths and does best on longer passages.
That is drawn from the abstract alone, so read the paper for the exact setup and numbers.
Language identification is commonly addressed using either neural architectures or statistical n-gram models. Neural approaches typically require substantial computational resources, whereas classical frequency-based methods offer efficient linear-time performance, but rely on distance metrics that are not always appropriate for compositional data. This work models character and bigram frequency distributions as compositional vectors constrained to the simplex and mapped via…
An Exploratory Study of Single Channel Surface Electromyography for Hand Gesture Classification
arXiv (cs.LG) · July 17, 2026Code-Poisoning Property Inference Attacks
arXiv (cs.AI) · July 17, 2026Sociocultural Influences on Opinion Formation: Word of Mouth Dynamics, Mass Media and Behavioural Development
arXiv (cs.CL) · July 17, 2026From Plausible to Actionable: A Position on LLM Self-Explanations