By identifying where reasoning first fails in each attempt, you can provide much richer training signal than outcome-only rewards—the key insight is that reasoning after a mistake is uninformative, so you only need to reward the correct prefix.
Cliff is a reward shaping strategy for training language models that identifies the first mistake in a reasoning chain and uses that to provide fine-grained feedback. Instead of just rewarding correct final answers, it marks where reasoning first goes wrong, then assigns positive rewards to the correct reasoning steps and negative rewards after the mistake.