Architectural innovations like sliding window attention and grouped-query attention let a 7B model punch far above its weight — Mistral proved that architecture matters as much as scale.
Introduces Mistral 7B, which outperforms Llama 2 13B on all benchmarks and Llama 1 34B on many, using grouped-query attention and sliding window attention for efficient inference with long sequences.