When deploying safety-critical ML systems with selective prediction, finite calibration data is the bottleneck—smart partition selection and error budget reallocation can recover 60% of the theoretical coverage gain, but naive approaches recover almost none.
This paper addresses how to safely deploy selective predictors (models that abstain when uncertain) by certifying they meet precision targets for specific user groups. The key challenge is that with limited calibration data, some groups may not have enough evidence to certify safety.