Training language models to follow instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang|March 4, 2022arXiv

Key Takeaway

RLHF alignment makes small models preferred over much larger unaligned ones — the recipe that made ChatGPT possible.

Summary

Introduces InstructGPT, which aligns GPT-3 to human intent using RLHF: supervised fine-tuning on demonstrations, reward model training from comparisons, and PPO optimization. The resulting 1.3B model is preferred over the 175B GPT-3.

training alignment