Small models can learn to strategically delegate hard problems to large models at the token level, cutting total inference costs dramatically while preserving reasoning quality—no separate router or LLM retraining needed.
PyroDash is a framework that pairs small language models with large language models for cost-efficient inference. During text generation, the small model learns when to ask the large model for help by emitting a control token, enabling a single handoff for completion. This approach reduces inference costs by up to 96% while maintaining competitive accuracy on math reasoning tasks.