Relaxed speculative decoding can speed up LLM inference but requires careful evaluation of capability trade-offs and works best with high-quality draft models—it's not a simple drop-in replacement for lossless speculative decoding.
This paper examines relaxed speculative decoding, a technique that speeds up LLM inference by allowing small deviations from the original model's output distribution. Unlike standard speculative decoding which preserves exact output probabilities, relaxed approaches trade some accuracy for faster generation.