2 research outputs found
External Validation of a Bayesian Network for Error Detection in Radiotherapy Plans
Artificial intelligence (AI) applications have recently been proposed to detect errors in radiotherapy plans. External validation of such systems is essential to assess their performance and safety before applying them to clinical practice. We collected data from 5238 patients treated at Maastro Clinic and introduced a range of common radiotherapy plan errors for the model to detect. We estimated the model's discrimination by calculating the area under the receiver-operating characteristic curve (AUC). We also assessed its clinical usefulness as an alert system that could reduce the need for manual checks by calculating the percentage of values flagged as errors and the positive predictive value (PPV) for a range of high sensitivities (95 %-99 %) and error prevalence. The AUC when considering all variables was 67.8% (95% CI, 65.6%-69.9%). The AUC varied widely for different types of errors (from 90.4% for table angle errors to 54.5% for planning tumor volume-PTV dose errors). The percentage of flagged values ranged from 84% to 90% for sensitivities between 95% and 99% and the PPV was only slightly higher than the prevalence of the errors. The model's performance in the external validation was significantly worse than that in its original setting (AUC of 68% versus 89%). Its usefulness as an alert system to reduce the need for manual checks is questionable due to the low PPV and high percentage of values flagged as potential errors to achieve a high sensitivity. We analyzed the apparent limitations of the model and we proposed actions to overcome them
Automatic quality assurance of radiotherapy treatment plans using Bayesian networks: A multi-institutional study
PurposeArtificial intelligence applications in radiation oncology have been the focus of study in the last decade. The introduction of automated and intelligent solutions for routine clinical tasks, such as treatment planning and quality assurance, has the potential to increase safety and efficiency of radiotherapy. In this work, we present a multi-institutional study across three different institutions internationally on a Bayesian network (BN)-based initial plan review assistive tool that alerts radiotherapy professionals for potential erroneous or suboptimal treatment plans. MethodsClinical data were collected from the oncology information systems in three institutes in Europe (Maastro clinic - 8753 patients treated between 2012 and 2020) and the United States of America (University of Vermont Medical Center [UVMMC] - 2733 patients, University of Washington [UW] - 6180 patients, treated between 2018 and 2021). We trained the BN model to detect potential errors in radiotherapy treatment plans using different combinations of institutional data and performed single-site and cross-site validation with simulated plans with embedded errors. The simulated errors consisted of three different categories: i) patient setup, ii) treatment planning and iii) prescription. We also compared the strategy of using only diagnostic parameters or all variables as evidence for the BN. We evaluated the model performance utilizing the area under the receiver-operating characteristic curve (AUC). ResultsThe best network performance was observed when the BN model is trained and validated using the dataset in the same center. In particular, the testing and validation using UVMMC data has achieved an AUC of 0.92 with all parameters used as evidence. In cross-validation studies, we observed that the BN model performed better when it was trained and validated in institutes with similar technology and treatment protocols (for instance, when testing on UVMMC data, the model trained on UW data achieved an AUC of 0.84, compared with an AUC of 0.64 for the model trained on Maastro data). Also, combining training data from larger clinics (UW and Maastro clinic) and using it on smaller clinics (UVMMC) leads to satisfactory performance with an AUC of 0.85. Lastly, we found that in general the BN model performed better when all variables are considered as evidence. ConclusionWe have developed and validated a Bayesian network model to assist initial treatment plan review using multi-institutional data with different technology and clinical practices. The model has shown good performance even when trained on data from clinics with divergent profiles, suggesting that the model is able to adapt to different data distributions