Here is a question that trips up a lot of students: if a model gives a grammatical sentence a higher probability than a broken one, does that mean it knows grammar? Not necessarily, because probability also tracks how common the words are, how plausible the meaning is, and what the model knows about the world. Grammaticality and likelihood get tangled together.

So rather than compare probabilities, the authors look inside the model. Using a technique called mass-mean probing, they check whether grammatical and ungrammatical sentences land in clearly separated regions of the model's internal representation space. They find that separation is robust across many pretrained models, holds up even after accounting for those confounding sentence properties, and generalizes across grammatical phenomena and, to a lesser degree, across languages.

This is drawn from the abstract, which cuts off partway, so read the paper for the methods and caveats.