An in-silico quality assurance study of contouring target volumes in thoracic tumors within a cooperative group setting

Introduction: Target delineation variability is a significant technical impediment in multi-institutional trials which employ intensity modulated radiotherapy (IMRT), as there is a real potential for clinically meaningful variances that can impact the outcomes in clinical trials. The goal of this study is to determine the variability of target delineation among participants from different institutions as part of Southwest Oncology Group (SWOG) Radiotherapy Committee\u27s multi-institutional in-silico quality assurance study in patients with Pancoast tumors as a dry run for trial implementation. Methods: CT simulation scans were acquired from four patients with Pancoast tumor. Two patients had simulation 4D-CT and FDG-FDG PET-CT while two patients had 3D-CT and FDG-FDG PET-CT. Seventeen SWOG-affiliated physicians independently delineated target volumes defined as gross primary and nodal tumor volumes (GTV_P and GTV_N), clinical target volume (CTV), and planning target volume (PTV).Six board-certified thoracic radiation oncologists were designated as the \u27Experts\u27 for this study. Their delineations were used to create a simultaneous truth and performance level estimation (STAPLE) contours using ADMIRE software (Elekta AB, Sweden 2017). Individual participants\u27 contours were then compared with Experts\u27 STAPLE contours. Results: When compared to the Experts\u27 STAPLE, GTV_P had the best agreement among all participants, while GTV_N showed the lowest agreement among all participants. There were no statistically significant differences in all studied parameters for all TVs for cases with 4D-CT versus cases with 3D-CT simulation scans. Conclusions: High degree of inter-observer variation was noted for all target volume except for GTV_P, unveiling potentials for protocol modification for subsequent clinically meaningful improvement in target definition. Various similarity indices exist that can be used to guide multi-institutional radiotherapy delineation QA credentialing

The impact of responding to patient messages with large language model assistance

Documentation burden is a major contributor to clinician burnout, which is rising nationally and is an urgent threat to our ability to care for patients. Artificial intelligence (AI) chatbots, such as ChatGPT, could reduce clinician burden by assisting with documentation. Although many hospitals are actively integrating such systems into electronic medical record systems, AI chatbots utility and impact on clinical decision-making have not been studied for this intended use. We are the first to examine the utility of large language models in assisting clinicians draft responses to patient questions. In our two-stage cross-sectional study, 6 oncologists responded to 100 realistic synthetic cancer patient scenarios and portal messages developed to reflect common medical situations, first manually, then with AI assistance. We find AI-assisted responses were longer, less readable, but provided acceptable drafts without edits 58% of time. AI assistance improved efficiency 77% of time, with low harm risk (82% safe). However, 7.7% unedited AI responses could severely harm. In 31% cases, physicians thought AI drafts were human-written. AI assistance led to more patient education recommendations, fewer clinical actions than manual responses. Results show promise for AI to improve clinician efficiency and patient care through assisting documentation, if used judiciously. Monitoring model outputs and human-AI interaction remains crucial for safe implementation.Comment: 4 figures and tables in main, submitted for revie

Machine Learning Applications in Head and Neck Radiation Oncology: Lessons From Open-Source Radiomics Challenges

Radiomics leverages existing image datasets to provide non-visible data extraction via image post-processing, with the aim of identifying prognostic, and predictive imaging features at a sub-region of interest level. However, the application of radiomics is hampered by several challenges such as lack of image acquisition/analysis method standardization, impeding generalizability. As of yet, radiomics remains intriguing, but not clinically validated. We aimed to test the feasibility of a non-custom-constructed platform for disseminating existing large, standardized databases across institutions for promoting radiomics studies. Hence, University of Texas MD Anderson Cancer Center organized two public radiomics challenges in head and neck radiation oncology domain. This was done in conjunction with MICCAI 2016 satellite symposium using Kaggle-in-Class, a machine-learning and predictive analytics platform. We drew on clinical data matched to radiomics data derived from diagnostic contrast-enhanced computed tomography (CECT) images in a dataset of 315 patients with oropharyngeal cancer. Contestants were tasked to develop models for (i) classifying patients according to their human papillomavirus status, or (ii) predicting local tumor recurrence, following radiotherapy. Data were split into training, and test sets. Seventeen teams from various professional domains participated in one or both of the challenges. This review paper was based on the contestants' feedback; provided by 8 contestants only (47%). Six contestants (75%) incorporated extracted radiomics features into their predictive model building, either alone (n = 5; 62.5%), as was the case with the winner of the “HPV” challenge, or in conjunction with matched clinical attributes (n = 2; 25%). Only 23% of contestants, notably, including the winner of the “local recurrence” challenge, built their model relying solely on clinical data. In addition to the value of the integration of machine learning into clinical decision-making, our experience sheds light on challenges in sharing and directing existing datasets toward clinical applications of radiomics, including hyper-dimensionality of the clinical/imaging data attributes. Our experience may help guide researchers to create a framework for sharing and reuse of already published data that we believe will ultimately accelerate the pace of clinical applications of radiomics; both in challenge or clinical settings

Test DICOM images set for HPV challenge

<p>This MD Anderson Cancer Center set of anonymized high-quality computed tomography (CT) scans with contrast represent a comparatively homogenous, uniform cohort of 315 oropharyngeal squamous carcinomas with detailed clinical history, consistent follow-up of > 2 years, known etiological/biological correlates (specifically, human papilloma virus status). Our major target is to assess/validate the radiomics workflow and predictive capacity of radiomics signatures from challenge participants.</p><p>We imported the CT scans from the patients’ electronic medical records, that were performed before the initiation of the radiation treatment course. All the patients were treated using the IMRT modality. Some patients were simultaneously prescribed chemotherapy. We intended that the CT films would be as much representative of the original simulation CT scans that were used for treatment planning, in which no contrast was injected according to our institutional policy.</p><p>Specifically, we posted one-half of the CT files from the dataset, in DICOM-RT format, on the Kaggle in Class server system, as a “training set”. DICOM-RT files were fully anonymized, with expert physician segmented primary tumor and lymph node regions of interest, to eliminate segmentation-related uncertainty for challengers. </p><p>The primary oropharyngeal tumor was segmented in red. Whereas, the metastatic cervical lymph nodes were segmented individually, rather than on the basis of the nodal level classification system. The green color was applied in contouring the nodes. </p><p>Both training and test sets include the following data for each DICOM-RT case:</p><ul><li>age</li><li>gender</li><li>race</li><li>tumor side and subsite</li><li>T-category</li><li>N-category</li><li>AJCC stage</li><li>Pathologic grade</li><li>smoking status (in pack-years)</li></ul><p>Challenge participants will also be able to download a “test" dataset, with the remaining random selected half of the dataset, which will have the HPV status blinded.</p

Training DICOM files set for Local control challenge

<p>This MD Anderson Cancer Center set of anonymized high-quality computed tomography (CT) scans with contrast represent a comparatively homogeneous, uniform cohort of 288 oropharynx cancer patients with detailed clinical history, consistent follow-up of > 2 years, known etiological/biological correlates (specifically, human papilloma virus status). Our major target is to assess/validate the radiomics workflow and predictive capacity of radiomics signatures from challenge participants.</p><p>We imported the CT scans from the patients’ electronic medical records, that were performed before the initiation of the radiation treatment course. All the patients were treated using the IMRT modality. Some patients were simultaneously prescribed chemotherapy. We intended that the CT films would be as much representative of the original simulation CT scans that were used for treatment planning, in which no contrast was injected according to our institutional policy.</p><p>Specifically, we posted around one-half of the CT scans from the dataset (138 patients), in DICOM-RT format, on the Kaggle in Class server system, as a “training set”. DICOM-RT files were fully anonymized, with expert physician segmenting primary tumor and lymph node as regions of interest, to eliminate segmentation-related uncertainty for challengers. </p><p>The primary oropharyngeal tumor was segmented in red. Whereas, the metastatic cervical lymph nodes were segmented individually, rather than on the basis of the nodal level classification system. </p><p>Both training and test sets include the following data for each DICOM-RT case:</p><ul><li>age</li><li>gender</li><li>race</li><li>tumor side and subsite</li><li>T-category</li><li>N-category</li><li>AJCC stage</li><li>Pathologic grade</li><li>smoking status (in pack-years)</li></ul><p>Challenge participants will also be able to download a “test" dataset, which includes the remaining randomly selected 150 patients' DICOM files and relevant clinical meta-data, with local control status blinded.Challenge participants will also be able to download a “test" dataset, which includes the remaining randomly selected half of the dataset, with local control status blinded.</p

Test set of clinical meta-data for Local Control Challenge

<p>This MD Anderson Cancer Center set of anonymized high-quality computed tomography (CT) scans with contrast represent a comparatively homogeneous, uniform cohort of 288 oropharynx cancer patients with detailed clinical history, consistent follow-up of > 2 years, known etiological/biological correlates (specifically, human papilloma virus status). Our major target is to assess/validate the radiomics workflow and predictive capacity of radiomics signatures from challenge participants.</p><p>We imported the CT scans from the patients’ electronic medical records, that were performed before the initiation of the radiation treatment course. All the patients were treated using the IMRT modality. Some patients were simultaneously prescribed chemotherapy. We intended that the CT films would be as much representative of the original simulation CT scans that were used for treatment planning, in which no contrast was injected according to our institutional policy.</p><p>Specifically, we posted around one-half of the CT scans from the dataset (138 patients), in DICOM-RT format, on the Kaggle in Class server system, as a “training set”. DICOM-RT files were fully anonymized, with expert physician segmenting primary tumor and lymph node as regions of interest, to eliminate segmentation-related uncertainty for challengers. </p><p>The primary oropharyngeal tumor was segmented in red. Whereas, the metastatic cervical lymph nodes were segmented individually, rather than on the basis of the nodal level classification system. </p><p>Both training and test sets include the following data for each DICOM-RT case:</p><ul><li>age</li><li>gender</li><li>race</li><li>tumor side and subsite</li><li>T-category</li><li>N-category</li><li>AJCC stage</li><li>Pathologic grade</li><li>smoking status (in pack-years)</li></ul><p>Challenge participants will also be able to download a “test" dataset, which includes the remaining randomly selected 150 patients' DICOM files and relevant clinical meta-data, with local control status blinded.Challenge participants will also be able to download a “test" dataset, which includes the remaining randomly selected half of the dataset, with local control status blinded.Challenge participants will also be able to download a “test" dataset, with the remaining random selected half of the dataset, which will have the local recurrence status blinded.</p

ReadMe for HPV Challenge

Supplemental information about the headings of the columns in the dataset

Test DICOM files set for Local control challenge

<p>This MD Anderson Cancer Center set of anonymized high-quality computed tomography (CT) scans with contrast represent a comparatively homogeneous, uniform cohort of 288 oropharynx cancer patients with detailed clinical history, consistent follow-up of > 2 years, known etiological/biological correlates (specifically, human papilloma virus status). Our major target is to assess/validate the radiomics workflow and predictive capacity of radiomics signatures from challenge participants.</p><p>We imported the CT scans from the patients’ electronic medical records, that were performed before the initiation of the radiation treatment course. All the patients were treated using the IMRT modality. Some patients were simultaneously prescribed chemotherapy. We intended that the CT films would be as much representative of the original simulation CT scans that were used for treatment planning, in which no contrast was injected according to our institutional policy.</p><p>Specifically, we posted around one-half of the CT scans from the dataset (138 patients), in DICOM-RT format, on the Kaggle in Class server system, as a “training set”. DICOM-RT files were fully anonymized, with expert physician segmenting primary tumor and lymph node as regions of interest, to eliminate segmentation-related uncertainty for challengers. </p><p>The primary oropharyngeal tumor was segmented in red. Whereas, the metastatic cervical lymph nodes were segmented individually, rather than on the basis of the nodal level classification system. </p><p>Both training and test sets include the following data for each DICOM-RT case:</p><ul><li>age</li><li>gender</li><li>race</li><li>tumor side and subsite</li><li>T-category</li><li>N-category</li><li>AJCC stage</li><li>Pathologic grade</li><li>smoking status (in pack-years)</li></ul><p>Challenge participants will also be able to download a “test" dataset, which includes the remaining randomly selected 150 patients DICOM files and relevant clinical meta-data, with local control status blinded.</p

Sample result submission for HPV Challenge

A sample submission file in the correct forma

Training set of clinical meta-data

<p>This MD Anderson Cancer Center set of anonymized high-quality computed tomography (CT) scans with contrast represent a comparatively homogenous, uniform cohort of 315 oropharyngeal squamous carcinomas with detailed clinical history, consistent follow-up of > 2 years, known etiological/biological correlates (specifically, human papilloma virus status). Our major target is to assess/validate the radiomics workflow and predictive capacity of radiomics signatures from challenge participants.</p><p>We imported the CT scans from the patients’ electronic medical records, that were performed before the initiation of the radiation treatment course. All the patients were treated using the IMRT modality. Some patients were simultaneously prescribed chemotherapy. We intended that the CT films would be as much representative of the original simulation CT scans that were used for treatment planning, in which no contrast was injected according to our institutional policy.</p><p>Specifically, we posted one-half of the CT files from the dataset, in DICOM-RT format, on the Kaggle in Class server system, as a “training set”. DICOM-RT files were fully anonymized, with expert physician segmented primary tumor and lymph node regions of interest, to eliminate segmentation-related uncertainty for challengers. </p><p>The primary oropharyngeal tumor was segmented in red. Whereas, the metastatic cervical lymph nodes were segmented individually, rather than on the basis of the nodal level classification system. The green color was applied in contouring the nodes. </p><p>Both training and test sets include the following data for each DICOM-RT case:</p><ul><li>age</li><li>gender</li><li>race</li><li>tumor side and subsite</li><li>T-category</li><li>N-category</li><li>AJCC stage</li><li>Pathologic grade</li><li>smoking status (in pack-years)</li></ul><p>Challenge participants will also be able to download a “test" dataset, with the remaining random selected half of the dataset, which will have the HPV status blinded.</p><div><br></div