BERT left performance on the table — training longer, on more data, with better hyperparameters matters more than architectural changes.
Shows that BERT was significantly undertrained and that careful tuning of hyperparameters, training data size, and training duration produces substantially better results. Removes next-sentence prediction and trains with larger batches and more data.