Skip to main content
CTOS 2025 Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.

Clinical Applications of AI

(P 165) EVALUATING LARGE LANGUAGE MODELS AS DECISION SUPPORT TOOLS IN MULTIDISCIPLINARY TUMOR BOARDS FOR GASTROINTESTINAL STROMAL TUMORS

    Author(s)

  • Melissa Harbrücker, MD (she/her/hers)

    Resident
    Sarcoma Unit, Department of Surgery, University Medical Centre Mannheim, Medical Faculty Mannheim, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

    • Co-Author(s)

  • Franka Menge, MD

    Dr.
    Sarcoma Unit, Department of Surgery, University Medical Centre Mannheim, Medical Faculty Mannheim, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

  • AT

    Arshia Taebi, Student

    Department of Surgery, University Medical Center Mannheim, Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany
    Mannheim, Baden-Wurttemberg, Germany

  • DN

    Dominik Nörenberg, Prof. Dr.

    Department of Radiology and Nuclear Medicine, University Medical Center Mannheim, Medical Faculty Mannheim, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

  • TS

    Tobias Speer, n/a

    Independent Researcher
    Ludwigshafen, Rheinland-Pfalz, Germany

  • CR

    Christoph Reissfelder, MD

    Department of Surgery, University Medical Centre Mannheim, Medical Faculty Mannheim, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

  • PH

    Peter Hohenberger, N/A, MD

    Sarcoma Unit, Department of Surgery, University Medical Centre Mannheim, Medical Faculty Mannheim, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

  • Cheng-Peng Li, MD (he/him/his)

    Attending Surgeon
    Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education/Beijing); Sarcoma Center
    Beijing, Beijing, China (People's Republic)

  • CY

    Cui Yang, MD

    Department of Surgery, Medical Faculty Mannheim, Mannheim School of Medicine, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

  • Jens Jakob, MD (he/him/his)

    Consultant Sarcoma Surgeon
    Sarcoma Unit, Department of Surgery, University Medical Centre Mannheim, Medical Faculty Mannheim, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

  • MH

    Madelaine Hettler, MD

    Sarcoma unit, Department of Surgery, University Medical Center and Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany.
    Mannheim, Baden-Wurttemberg, Germany

Objective: To assess the concordance of treatment recommendations generated by two reasoning large language models (LLMs), OpenAI o1 and DeepSeek-R1, with decisions made by a multidisciplinary team (MDT) in real-world gastrointestinal stromal tumor (GIST) cases, and to evaluate the feasibility of using LLMs as evaluators in clinical oncology.

Methods: In a retrospective single-center study, 65 real-world GIST cases presented at a certified sarcoma MDT were selected from a total of 753 cases. Two advanced LLMs were prompted to simulate a tumor board discussion using a multi-role prompting strategy, supported by the German S3 sarcoma guidelines. Recommendations were independently rated by five clinical experts using a 5-point Likert scale. Additionally, Mistral AI served as an automated evaluator. Recommendations were assessed for concordance with MDT decisions and alignment with guidelines.

Results: OpenAI o1 demonstrated higher concordance with MDT decisions (76.9% vs. 53.8%, p < 0.001) and more outputs aligned with MDT or guidelines (80.0% vs. 63.1%, p = 0.005) compared to DeepSeek-R1. Expert inter-rater reliability was excellent (ICC = 0.929). Mistral AI ratings moderately aligned with expert consensus (κ = 0.588). OpenAI o1 outputs were significantly more verbose (mean 1359 vs. 697 characters, p < 0.001), offering detailed justifications that may support documentation but present usability challenges. The LLM judge also detected more instances of hallucinations in DeepSeek-R1 outputs.

Conclusion: Reasoning LLMs, particularly OpenAI o1, show promising performance as decision support tools in complex GIST cases and demonstrate potential for partially automating clinical evaluation through LLM-as-a-judge frameworks. Their deployment should be supervised by experts to ensure reliability. One limitation of the study is that a large number of MTBs were evaluated, but only from a single center. Further multicenter studies and integration into real-time MDT workflows are warranted.

Case descriptions
Upload Image