You can speed up LLM inference by skipping less important values in matrix operations—attention layers benefit most from this, while MLPs are harder to compress—and the speedups scale better on longer inputs.
This paper introduces Reduced Matrix Multiplication (RMM), a training-free method that speeds up language model inference by selectively removing less important numerical values from matrix multiplications. The approach works without changing model weights and maintains accuracy while reducing computation, with benefits increasing at longer sequence lengths.