(P 168) LEVERAGING LARGE LANGUAGE MODELS FOR STRUCTURED INFORMATION EXTRACTION FROM EWING SARCOMA PATHOLOGY REPORTS

Objective: Risk stratification in Ewing sarcoma (EWS) is dependent on tumor stage and size, with new prognostic factors needed. We will address this knowledge gap by leveraging computational pathology and advances in machine learning (ML). For this report we address the labor-intensive and time-consuming task to build a case bank of structured data from pathology reports to allow ML based knowledge discovery.

Methods: We secured 1,018 digitized histology images at time of biopsy with institutional pathology reports, from 932 unique patients enrolled on 7 Children’s Oncology Group EWS clinical trials (AEWS0031, AEWS1031, AEWS1221, B947, AEWS02B1, AEWS07B1 and APEC14B1). Pathology reports were converted to text files using optical character recognition and we performed exploratory studies for pathological attributes for study with large language models (LLM).

Results: We developed a potential critical variable and EWS variable list based on the College of American Pathologists protocol on pathology reporting in JavaScript object notation form. Attributes such as procedure, specimen, and tumor site as well as their sub-attributes present broadly in pathology reports, while lympho-vascular invasion and regional lymph node status were present in over 80% of the reports. Accurate data extraction challenges include variability in reporting structure and terms, as well as the risk of hallucinations (which are LLM generated content that appear plausible but are factually false). To address these risks and optimize accuracy and reliability we use multiple LLMs to estimate pathological attributes, then combine the outputs by using ensemble methods, such as majority vote. The extracted structured data will enrich the context for classification tasks with computational pathology for this rare disease.

Conclusion: We have successfully extracted pathology report variables using LLM from a rare sarcoma. We will now merge AI extracted histology features, and pathology report variables with known clinical outcomes to create an integrated risk stratification model for sarcoma patients that will be testable in future prospective clinical trials.