By treating server refusals as feedback signals rather than failures, you can build adaptive request dispatchers that automatically track available capacity without predicting output lengths or manually tuning rates.
Cap-and-Track is a system for efficiently routing inference requests to LLM serving clusters that dynamically adjusts to changing capacity. Instead of using fixed dispatch rates, it uses refusal signals from overloaded servers to probe the current capacity ceiling and adjust request routing in real-time, reducing GPU requirements by 43% while maintaining quality.