Implicit Representations of Grammaticality in Language Models

Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu, Roger P. Levy, Yoon Kim|May 6, 2026arXiv

Key Takeaway

Language models learn grammaticality as a distinct concept from string probability, hidden in their internal representations rather than reflected in output probabilities—you can extract this knowledge with a simple linear probe.

Summary

Language models generate grammatical text but their probability scores don't clearly distinguish grammatical from ungrammatical sentences.

evaluation

Key Terms

linear-probes internal-representations syntactic-correctness cross-lingual-generalization