BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee|October 11, 2018arXiv

Key Takeaway

Bidirectional pre-training on unlabeled text followed by task-specific fine-tuning created a universal approach to NLP that obsoleted most task-specific architectures.

Summary

Introduces BERT, a bidirectional transformer pre-trained on masked language modeling and next-sentence prediction. Fine-tuning BERT achieves state-of-the-art results on 11 NLP tasks simultaneously, establishing transfer learning as the dominant paradigm.

architecture training