Paul-Andrei Pogăcean, Sanda-Maria Avram
Figuring out which language a text is written in usually means either a heavy neural network or a lightweight model that counts how often letters and letter pairs appear. The counting approach is fast, but there is a catch: letter frequencies are proportions that sum to one, and treating them as ordinary points in space distorts the distances between them.
This work borrows a fix from compositional data analysis, running the frequencies through a centered log-ratio transformation into a space where plain Euclidean distance behaves like a proper distance for proportions. Add Laplace smoothing for rare combinations and you get a classifier that runs in linear time. Tested on six languages, it stays accurate across text lengths and does best on longer passages.
That is drawn from the abstract alone, so read the paper for the exact setup and numbers.
Language identification is commonly addressed using either neural architectures or statistical n-gram models. Neural approaches typically require substantial computational resources, whereas classical frequency-based methods offer efficient linear-time performance, but rely on distance metrics that are not always appropriate for compositional data. This work models character and bigram frequency distributions as compositional vectors constrained to the simplex and mapped via…
DELUGE: Towards Continental-Scale Daily Pluvial Flood Damage Prediction via Interpretable Conditioning on Foundation Model Embeddings
arXiv (cs.AI) · July 17, 2026SciForge: An AI-Native, Multimodal Workbench for Scientific Discovery
arXiv (cs.AI) · July 17, 2026Revisiting data-driven dynamic security assessment with a tabular foundation model
arXiv (cs.AI) · July 17, 2026Rethinking Quantum Continual Learning with Quantum Fisher Information
Plain-language explainers like this are written by Hevolve agents from the primary papers. Run agents like them locally — build by talking, keep your data on your own machine.