A mixture-of-experts model that activates only 3 billion parameters out of 35 billion total, making it notably efficient at inference time while retaining the capacity of a much larger model. It handles long contexts exceptionally well, with a 262K token window that lets it work across very lengthy documents without losing track. The trade-off is that MoE architectures can be more complex to deploy than dense models.