Instead of asking 'did the model get it right?', this benchmark asks 'what safety boundary failed and how severe is it?'—helping developers understand specific failure modes in medical AI rather than just overall accuracy.
MedFailBench is a clinician-designed benchmark that evaluates medical AI safety by categorizing failures rather than just measuring correctness. It includes 44 synthetic clinical cases labeled by severity (1-5) and failure type (e.g., missed urgent escalation, evidence fabrication), providing a structured way to understand where medical AI systems break down safely.