A pre-trained language model that learns by predicting which tokens in a sentence have been replaced, making it efficient and effective for downstream tasks.
World knowledge accuracy, recall of facts and relationships