Designing model architecture around deployment constraints (CPU, 4-bit weights, fixed memory) from the start beats the conventional approach of building large then compressing—hybrid convolution-attention architectures can match attention-only models while being significantly faster at inference.
This paper presents Daedalus-150M, a small language model designed specifically for CPU inference by combining convolutions and attention strategically. Rather than shrinking a large model, the authors built from scratch with CPU constraints in mind, using full attention in only 6 of 18 blocks while the rest use short convolutions with fixed memory.