Jointly optimizing compression and adaptation using task-aware subspaces beats the standard two-step approach, delivering better accuracy with fewer parameters on both vision and language models.
JACTUS combines model compression and task adaptation in a single step rather than doing them sequentially. Instead of compressing a model first and then fine-tuning it, the method estimates what directions matter for your specific task and compresses the model while preserving those directions.