By assuming sparse reward structure, you can learn from human preferences with differential privacy guarantees that depend only on the number of relevant features (s*), not the full model dimension (d)—making privacy practical for large language models.
This paper tackles privacy-preserving reward learning for aligning language models with human feedback. The key innovation is exploiting sparsity in reward models—assuming only a small subset of features matter—to achieve differential privacy guarantees that don't scale with model size.