Deep Reinforcement Learning from Human Preferences

Paul F. Christiano, Jan Leike, Tom B. Brown|June 12, 2017arXiv

Key Takeaway

Human preference comparisons can replace hand-crafted reward functions for training RL agents — the conceptual ancestor of RLHF for language models.

Summary

Demonstrates that complex RL behaviors can be learned from human preferences rather than hand-engineered reward functions. Non-expert humans provide comparisons between pairs of agent behaviors, and a reward model is trained from these comparisons.

training alignment