(P 174) BIAS OR BEST FIT? A COMPARATIVE ANALYSIS OF SEER AND NCDB DATASETS IN SINGLE MODEL MACHINE LEARNING FOR PREDICTING OSTEOSARCOMA SURVIVAL OUTCOMES

Objective: Machine learning models have been increasingly utilized in orthopaedic oncology to predict survival outcomes for osteosarcoma patients. Typically, these models are trained on a single dataset, such as the Surveillance, Epidemiology, and End Results(SEER) or the National Cancer Database(NCDB). However, this approach may lead to biased predictions that mirror the characteristics of the training data. In this study, we aimed to determine which dataset performs better with a single-model machine learning(SMML) model.

Methods: We included a total of 4,049 patients from the NCDB dataset and 4,278 patients from the SEER dataset with confirmed osteosarcoma. We developed an SMML model for each dataset and compared the performance of these models. The primary outcomes to assess each SMML were accuracy, area under the receiver curve(AUC), Brier score, precision, and F1 score.

Results: SMML models trained and validated on the same dataset displayed excellent performance predicting 2-year and 5-year overall survival(OS). The NCDB-trained/validated model had an AUC of 0.927 at 2 years and 0.911 at 5 years, compared to 0.898 and 0.916 for the SEER-trained/validated model. The NCDB-trained and SEER-validated model had an AUC of 0.665 and 0.599 for 2-year and 5-year OS, respectively. The SEER-trained and NCDB-validated model had an AUC of 0.606 and 0.563 for 2-year and 5-year OS, respectively.

Conclusion: These results highlight the challenges in generalizing models across different datasets in medical machine learning development and prognosis prediction, suggesting significant differences between the NCDB and SEER datasets or potential overfitting to dataset-specific characteristics.

Comparison of performance metrics between single model same dataset training and validation and cross dataset validation