OPT: Open Pre-trained Transformer Language Models

Susan Zhang, Stephen Roller, Naman Goyal|May 2, 2022arXiv

Key Takeaway

Meta released a full GPT-3-class model openly — OPT was one of the first major efforts to democratize access to large language model research.

Summary

Releases OPT-175B, a suite of decoder-only transformers from 125M to 175B parameters, with full model weights and training code. Aims to enable reproducible research on large language models by matching GPT-3 performance openly.

scaling training