Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination

Key Takeaway

Open medical LLMs achieve competitive exam scores but lack reliable confidence signals, sound clinical reasoning for many correct answers, and don't actually use medical images—critical gaps for real healthcare deployment beyond raw accuracy metrics.