Combining structured scene graphs with retrieval-augmented generation substantially improves surgical scene captioning accuracy and factual correctness, suggesting that domain-specific structured representations can enhance LLM-based medical documentation.
GRAC is a system that automatically generates descriptions of surgical scenes by combining graph representations of surgical elements (instruments, actions, anatomy) with retrieval-augmented generation. It retrieves similar past surgical scenes and uses them to guide caption generation, achieving significantly better descriptions than existing methods on surgical video benchmarks.