Autoregressive pre-training on raw text creates general-purpose language representations — the first GPT model and the beginning of the scaling paradigm.
Introduces GPT-1, demonstrating that generative pre-training of a transformer decoder on unlabeled text followed by discriminative fine-tuning produces strong performance across diverse NLP tasks with minimal task-specific architecture changes.