LLMs can reliably detect major flaws in black-box clinical models through query access alone, but their reports of problems and new discoveries require verification against the actual model and real data—they generate many false positives.
Researchers tested whether GPT-4 Astra can audit clinical prediction models by querying them and reading predictions. They trained models on stroke data, planted 24 deliberate flaws, and had the LLM find them blind.