Knowledge distillation can dramatically compress transformer models with minimal quality loss — a template for making large models practical for deployment.
Applies knowledge distillation to BERT, producing a model 60% smaller and 60% faster while retaining 97% of BERT performance. Demonstrates that much of a large model capacity can be compressed into a smaller one.