Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, Adam Roberts|October 23, 2019arXiv

Key Takeaway

Casting all NLP tasks as text-to-text unifies the approach to transfer learning and enables systematic comparison of design choices at scale.

Summary

Proposes T5, which frames every NLP task as a text-to-text problem. Systematically studies pre-training objectives, architectures, data, and scale, establishing best practices for transfer learning with transformers.

architecture training scaling