Requential coding compresses models by measuring disagreement between teacher and student rather than counting parameters, producing dramatically shorter codes that improve with scale and enabling state-of-the-art generalization guarantees for billion-parameter LLMs.
This paper introduces requential coding, a compression method that measures how well a student model learns by having a teacher model select training samples from the student's own distribution.