(P 166) LARGE LANGUAGE MODEL EXTRACTION OF STRUCTURED DATA IN UNSTRUCTURED SARCOMA PATHOLOGY NOTES: A PILOT STUDY

Objective: Bone and soft-tissue sarcomas are rare, heterogeneous tumors with unique treatment challenges. This heterogeneity has led to difficulty with registry curation, as well as effective outcomes assessment. The purpose of this study was to assess the feasibility of LLMs in extracting key information from surgical pathology notes following surgical sarcoma excision.

Methods: Surgical pathology notes were retrospectively identified from 353 patients who received surgical intervention for a diagnosis of bone or soft tissue sarcoma at one institution from 2000-2024. Data points of interest included tumor diagnosis (osteosarcoma, Ewing’s sarcoma, etc), greatest dimension (cm), closest margin (cm), and grade. All labeled pathology notes were then split into test and training sets in an 85/15 split (303 testing notes, 50 training notes). Each LLM data extraction pipeline for each data point was developed using Llama-3-8B-Instruct (Meta).

Results: All LLM-enabled data extraction pipelines demonstrated satisfactory accuracies on the testing set following iterative prompt modification. Diagnosis, greatest dimension, closest margin, and grade accuracies were 98%, 88%, 85%, and 94%, respectively. The mean time of data extraction for each data point was 4.1 seconds.

Conclusion: This pilot study demonstrates critical proof of concept for automated registry production utilizing LLM outputs of unstructured pathology note data.