Skip to main content
CTOS 2025 Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.

Clinical Applications of AI

(P 169) BENCHMARKING LARGE LANGUAGE MODELS VS MULTIDISCIPLINARY TUMOR BOARDS OF 21 SARCOMA CENTERS

    Author(s)

  • Jens Jakob, MD (he/him/his)

    Consultant Sarcoma Surgeon
    Sarcoma Unit, Department of Surgery, University Medical Centre Mannheim, Medical Faculty Mannheim, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

    • Co-Author(s)

  • Cheng-Peng Li, MD (he/him/his)

    Attending Surgeon
    Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education/Beijing); Sarcoma Center
    Beijing, Beijing, China (People's Republic)

  • AK

    Aimé Terence Kalisa, Cand. Med.

    Department of Surgery, Medical Faculty Mannheim, Mannheim School of Medicine, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

  • SR

    Siyer Roohani, MD

    Department of Radiation Oncology, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin
    Berlin, Berlin, Germany

  • KH

    Kamal Hummedah, MD

    Department of Surgery, Medical Faculty Mannheim, Mannheim School of Medicine, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

  • CR

    Christoph Reissfelder, MD

    Department of Surgery, University Medical Centre Mannheim, Medical Faculty Mannheim, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

  • CY

    Cui Yang, MD

    Department of Surgery, Medical Faculty Mannheim, Mannheim School of Medicine, Heidelberg University
    Mannheim, Baden-Wurttemberg, Germany

Objective: This study compares treatment recommendations of four leading large language models (LLMs) with those of 21 multidisciplinary tumor boards (MDTs) from certified sarcoma centers.

Methods: We simulated STS MTBs using four LLMs – Llama 3.2-vison: 90b, Claude 3.5 Sonnet, DeepSeek-R1, and OpenAI-o1 – based on five anonymized STS cases (inguinal dedifferentiated liposarcoma (G2), chest wall myxoid liposarcoma (G3), retroperitoneal dedifferentiated liposarcoma (G2), well-differentiated liposarcoma (G1), popliteal pleomorphic rhabdomyosarcoma (G3)). Each model was queried 21 times per case using standardized input, and the responses were compared with human MTBs for consistency within the model, agreement of treatment recommendations, and source citations. The presented cases had previously been discussed in a ring trial of sarcoma MDTs at certified German sarcoma centers.

Results: LLMs showed consistency between models in 20% of cases, and their recommendations were consistent with human consensus patterns in 20–60% of cases. The best model, Claude 3.5 Sonnet, agreed with the experts in 60% of cases. MTD recommendations were very heterogeneous, which contextualizes the variable performance of the LLMs (figure 1). Some common human recommendations were missing from the LLM outputs. LLMs used national and international guidelines as sources for recommendations (e.g., the German sarcoma guideline in 24.8% to 55.2% of responses).

Conclusion: Both the recommendation patterns in the sarcoma ring study and those of the AI tools show considerable heterogeneity in the recommendations for a specific sarcoma case. The main reason for this is that there are only a few randomized studies available for common treatment decisions and that the guidelines do not provide clear, unambiguous recommendations. The considerable disagreements highlight the current limitations in the training and evaluation of AI tools as well as the heterogeneity of clinical practice in various sarcoma centers.

Pattern of treatment recommendations for a 67-year-ole female patient with a high grade myxoid liposarcoma of the chest wall for MTDs of 21 different sarcoma centers and 21 consecutive runs of various LLMs.
Upload Image