Artificial General Intelligence for Radiation Oncology

The emergence of artificial general intelligence (AGI) is transforming radiation oncology. As prominent vanguards of AGI, large language models (LLMs) such as GPT-4 and PaLM 2 can process extensive texts and large vision models (LVMs) such as the Segment Anything Model (SAM) can process extensive imaging data to enhance the efficiency and precision of radiation therapy. This paper explores full-spectrum applications of AGI across radiation oncology including initial consultation, simulation, treatment planning, treatment delivery, treatment verification, and patient follow-up. The fusion of vision data with LLMs also creates powerful multimodal models that elucidate nuanced clinical patterns. Together, AGI promises to catalyze a shift towards data-driven, personalized radiation therapy. However, these models should complement human expertise and care. This paper provides an overview of how AGI can transform radiation oncology to elevate the standard of patient care in radiation oncology, with the key insight being AGI's ability to exploit multimodal clinical data at scale

Multimodality Imaging in Prostate Cancer

ABSTRACT Prostate cancer is the most common cancer in men in Finland. Its aggressiveness varies widely, from indolent to fatal disease. Accurate characterization of prostate cancer is extremely essential to prevent overtreatment while sustaining good survivorship and high quality of life. This is feasible using novel technology in imaging and automatic tools in treatment planning. In the first part of this thesis work, the aim was to evaluate anti-1-amino-3-18Ffluorocyclobutane-1-carboxylic acid (18F-FACBC) PET/CT, PET/MRI, and multiparametric MRI (mpMRI) in detection of primary prostate cancer. The uptake of 18F-FACBC was significantly stronger in tumors with higher Gleason score and it may therefore assist in targeted biopsies when combined with MRI. 18F-FACBC PET/MRI outperformed PET/CT but did not demonstrate higher diagnostic performance than mpMRI performed separately. Furthermore, PET/MRI and mpMRI failed to detect pelvic lymph node metastasis measuring less than 8mm. 18F-FACBC PET/MRI is promising in characterization of primary prostate cancer, especially if ablative treatments are planned. It is not likely to replace mpMRI in clinical practice. The second study assessed multimodality imaging in detecting bone metastasis in high-risk prostate cancer and breast cancer patients. All patients underwent 99mTc-HDP bone scintigraphy (BS), 99mTc-HDP SPECT, 99mTc-HDP SPECT/CT, 18F-NaF PET/CT, and whole body (wb) MRI+DWI. 99mTc-HDP SPECT/CT, 18F-NaF PET/CT, and wbMRI+DWI had superior sensitivity compared to conventional nuclear imaging. In particular non-BS techniques showed less equivocal findings. wbMRI+DWI was as accurate as 18F-NaF PET/CT for detecting bone metastasis and may be considered a potential “single-step” imaging modality for detection of bone metastasis in high-risk patients with prostate and breast cancer. The purpose of the third study was to evaluate and validate the performance of a fully automated segmentation tool (AST) in MRI-based radiotherapy planning of prostate cancer. It showed high agreement for delineating prostate, bladder, and rectum, compared to manual contouring, and suggested adoption of AST in clinical practice. Finally, the fourth study investigated the long-term toxicity after biologically guided radiotherapy in men with localized prostate cancer. Carbon-11 acetate (11C-ACE) PET-CT was used to guide dose escalation into metabolically active intraprostatic lesions. 11C-ACE PET-guided radiotherapy was feasible and well tolerated. Although erectile dysfunction was relatively common, severe gastro-intestinal symptoms were very rare, and no grade 3 genitourinary symptoms were present at five years after radiotherapy. The findings of this thesis have potential to improve diagnostic imaging and radiotherapy planning in primary and metastatic prostate cancer. Eventually, they are likely to improve patients’ quality of life and survival. KEYWORDS: prostate cancer, magnetic resonance imaging, positron emission tomography, radiotherapy planning, toxicity, bone metastasisTIIVISTELMÄ Eturauhassyöpä on miesten yleisin syöpä Suomessa. Sen taudinkuva vaihtelee laajasti rauhallisesta aggressiiviseen ja tappavaan. On oleellista, että taudin luonne arvioidaan tarkasti, jotta vältytään sen liialliselta hoidolta, tinkimättä erinomaisista hoitotuloksista selviytymisessä ja elämän laadussa. Uudet kuvantamisteknologiat ja automaattityökalut mahdollistavat tämän. Tämän väitöskirjan ensimmäisessä osatyössä oli tavoitteena arvioida anti-1-amino-3-18Ffluorosyklobutaani-1-karboksyylihappo (18F-FACBC) PET-tietokonetomografiaa (TT), PET-magneettiresonanssikuvantamista (MRI) ja multiparametrista MRI-kuvantamista (mpMRI) eturauhassyövän diagnoosivaiheessa. 18F-FACBC-kertymät olivat tilastollisesti merkitsevästi voimakkaampia korkean Gleason-luokituksen kasvaimissa, joten yhdistettyä PET-MRI-kuvantamista voidaan käyttää hyväksi esimerkiksi kohdennetussa koepalojen otossa. 18F-FACBC PET-MRI oli parempi kuin PET-TT ja samanveroinen kuin mpMRI eturauhassyövän diagnostiikassa. PET-MRI ja mpMRI eivät havainneet alle 8 mm:n läpimittaisia imusolmukemetastaaseja. 18F-FACBC PET-MRI on lupaava kuvantamismuoto eturauhassyövän diagnostiikassa, erityisesti kajoavia hoitoja suunniteltaessa, mutta ei korvanne mpMRI:a kliinisessä käytössä. Toinen osatyö käsitteli luustoetäpesäkkeiden toteamista eri kuvantamismenetelmillä korkean uusiutumisriskin eturauhas- ja rintasyöpäpotilailla. Kaikille potilaille tehtiin 99mTc-HDP luustokarttakuvaus, 99mTc-HDP SPECT, 99mTc-HDP SPECT-TT, 18F-NaF PET-TT ja koko kehon MRI diffuusiopainotettuna (wbMRI+DWI). 99mTc-HDP SPECT-TT, 18F-NaF PET-TT ja wbMRI+DWI olivat perinteistä luustokarttaa herkempiä luustometastaasien toteamisessa, koska epäspesifeiksi määriteltyjä muutoksia oli vähemmän. wbMRI+DWI osoitti yhtäläistä tarkkuutta luustometastaasien diagnosoinnissa 18F-NaF PET-TT:n verrattuna, joten sitä voitaisiin hyödyntää, käytettäessä vain yhtä kuvantamistapaa näiden potilaiden luustometastaasien toteamiseen. Kolmas osatyö arvioi ja validoi täysin automaattisen piirtotyökalun käyttöä MRI-pohjaisen sädehoidon suunnittelussa eturauhassyöpäpotilailla. Työkalu suoriutui hyvin eturauhasen, virtsarakon ja peräsuolen rajauksesta asiantuntijan käsin tekemiin rajauksiin verrattuna, puoltaen työkalun käyttöä luotettavasti myös kliinisessä työssä. Viimeisenä, neljännessä osatyössä arvioitiin biologisesti ohjatun eturauhassyövän sädehoidon aiheuttamia pitkäaikaishaittoja. Hiili-11 asetaatti (11C-ACE) PET-TT-kuvantamisen avulla suunniteltiin sädehoito, jossa metabolisesti aktiivisiin eturauhasen sisäisiin muutoksiin kohdistettiin korkeammat sädeannokset. 11C-ACE-PET-TT-ohjattu sädehoito oli toteuttamiskelpoinen ja hyvin siedetty. Vaikka erektiohäiriöt olivat suhteellisen yleisiä, vakavat suoliston haittavaikutukset olivat hyvin harvinaisia, eikä kolmannen asteen virtsateiden haittavaikutuksia esiintynyt lainkaan viiden vuoden kuluttua sädehoidosta. Tämän väitöskirjan löydökset voivat parantaa eturauhassyövän primaaridiagnostiikan kuvantamista ja sädehoidon suunnittelua, sekä luustoetäpesäkkeiden diagnostiikkaa. Näin voidaan kohentaa potilaiden elämänlaatua ja selviytymistä. AVAINSANAT: Eturauhassyöpä, magneettikuvaus, positroniemissiotomografia, sädehoidon suunnittelu, haittavaikutukset, luuston etäpesäkkee

Multiparametric Magnetic Resonance Imaging Artificial Intelligence Pipeline for Oropharyngeal Cancer Radiotherapy Treatment Guidance

Oropharyngeal cancer (OPC) is a widespread disease and one of the few domestic cancers that is rising in incidence. Radiographic images are crucial for assessment of OPC and aid in radiotherapy (RT) treatment. However, RT planning with conventional imaging approaches requires operator-dependent tumor segmentation, which is the primary source of treatment error. Further, OPC expresses differential tumor/node mid-RT response (rapid response) rates, resulting in significant differences between planned and delivered RT dose. Finally, clinical outcomes for OPC patients can also be variable, which warrants the investigation of prognostic models. Multiparametric MRI (mpMRI) techniques that incorporate simultaneous anatomical and functional information coupled to artificial intelligence (AI) approaches could improve clinical decision support for OPC by providing immediately actionable clinical rationale for adaptive RT planning. If tumors could be reproducibly segmented, rapid response could be classified, and prognosis could be reliably determined, overall patient outcomes would be optimized to improve the therapeutic index as a function of more risk-adapted RT volumes. Consequently, there is an unmet need for automated and reproducible imaging which can simultaneously segment tumors and provide predictive value for actionable RT adaptation. This dissertation primarily seeks to explore and optimize image processing, tumor segmentation, and patient outcomes in OPC through a combination of advanced imaging techniques and AI algorithms. In the first specific aim of this dissertation, we develop and evaluate mpMRI pre-processing techniques for use in downstream segmentation, response prediction, and outcome prediction pipelines. Various MRI intensity standardization and registration approaches were systematically compared and benchmarked. Moreover, synthetic image algorithms were developed to decrease MRI scan time in an effort to optimize our AI pipelines. We demonstrated that proper intensity standardization and image registration can improve mpMRI quality for use in AI algorithms, and developed a novel method to decrease mpMRI acquisition time. Subsequently, in the second specific aim of this dissertation, we investigated underlying questions regarding the implementation of RT-related auto-segmentation. Firstly, we quantified interobserver variability for an unprecedented large number of observers for various radiotherapy structures in several disease sites (with a particular emphasis on OPC) using a novel crowdsourcing platform. We then trained an AI algorithm on a series of extant matched mpMRI datasets to segment OPC primary tumors. Moreover, we validated and compared our best model\u27s performance to clinical expert observers. We demonstrated that AI-based mpMRI OPC tumor auto-segmentation offers decreased variability and comparable accuracy to clinical experts, and certain mpMRI input channel combinations could further improve performance. Finally, in the third specific aim of this dissertation, we predicted OPC primary tumor mid-therapy (rapid) treatment response and prognostic outcomes. Using co-registered pre-therapy and mid-therapy primary tumor manual segmentations of OPC patients, we generated and characterized treatment sensitive and treatment resistant pre-RT sub-volumes. These sub-volumes were used to train an AI algorithm to predict individual voxel-wise treatment resistance. Additionally, we developed an AI algorithm to predict OPC patient progression free survival using pre-therapy imaging from an international data science competition (ranking 1st place), and then translated these approaches to mpMRI data. We demonstrated AI models could be used to predict rapid response and prognostic outcomes using pre-therapy imaging, which could help guide treatment adaptation, though further work is needed. In summary, the completion of these aims facilitates the development of an image-guided fully automated OPC clinical decision support tool. The resultant deliverables from this project will positively impact patients by enabling optimized therapeutic interventions in OPC. Future work should consider investigating additional imaging timepoints, imaging modalities, uncertainty quantification, perceptual and ethical considerations, and prospective studies for eventual clinical implementation. A dynamic version of this dissertation is publicly available and assigned a digital object identifier through Figshare (doi: 10.6084/m9.figshare.22141871)

Incorporating Cardiac Substructures Into Radiation Therapy For Improved Cardiac Sparing

Growing evidence suggests that radiation therapy (RT) doses to the heart and cardiac substructures (CS) are strongly linked to cardiac toxicities, though only the heart is considered clinically. This work aimed to utilize the superior soft-tissue contrast of magnetic resonance (MR) to segment CS, quantify uncertainties in their position, assess their effect on treatment planning and an MR-guided environment. Automatic substructure segmentation of 12 CS was completed using a novel hybrid MR/computed tomography (CT) atlas method and was improved upon using a 3-dimensional neural network (U-Net) from deep learning. Intra-fraction motion due to respiration was then quantified. The inter-fraction setup uncertainties utilizing a novel MR-linear accelerator were also quantified. Treatment planning comparisons were performed with and without substructure inclusions and methods to reduce radiation dose to sensitive CS were evaluated. Lastly, these described technologies (deep learning U-Net) were translated to an MR-linear accelerator and a segmentation pipeline was created. Automatic segmentations from the hybrid MR/CT atlas was able to generate accurate segmentations for the chambers and great vessels (Dice similarity coefficient (DSC) \u3e 0.75) but coronary artery segmentations were unsuccessful (DSC\u3c0.3). After implementing deep learning, DSC for the chambers and great vessels was ≥0.85 along with an improvement in the coronary arteries (DSC\u3e0.5). Similar accuracy was achieved when implementing deep learning for MR-guided RT. On average, automatic segmentations required ~10 minutes to generate per patient and deep learning only required 14 seconds. The inclusion of CS in the treatment planning process did not yield statistically significant changes in plan complexity, PTV, or OAR dose. Automatic segmentation results from deep learning pose major efficiency and accuracy gains for CS segmentation offering high potential for rapid implementation into radiation therapy planning for improved cardiac sparing. Introducing CS into RT planning for MR-guided RT presented an opportunity for more effective sparing with limited increase in plan complexity

Organ-focused mutual information for nonrigid multimodal registration of liver CT and Gd–EOB–DTPA-enhanced MRI

Accurate detection of liver lesions is of great importance in hepatic surgery planning. Recent studies have shown that the detection rate of liver lesions is significantly higher in gadoxetic acid-enhanced magnetic resonance imaging (Gd–EOB–DTPA-enhanced MRI) than in contrast-enhanced portal-phase computed tomography (CT); however, the latter remains essential because of its high specificity, good performance in estimating liver volumes and better vessel visibility. To characterize liver lesions using both the above image modalities, we propose a multimodal nonrigid registration framework using organ-focused mutual information (OF-MI). This proposal tries to improve mutual information (MI) based registration by adding spatial information, benefiting from the availability of expert liver segmentation in clinical protocols. The incorporation of an additional information channel containing liver segmentation information was studied. A dataset of real clinical images and simulated images was used in the validation process. A Gd–EOB–DTPA-enhanced MRI simulation framework is presented. To evaluate results, warping index errors were calculated for the simulated data, and landmark-based and surface-based errors were calculated for the real data. An improvement of the registration accuracy for OF-MI as compared with MI was found for both simulated and real datasets. Statistical significance of the difference was tested and confirmed in the simulated dataset (p < 0.01)

Incorporating Cardiac Substructures Into Radiation Therapy For Improved Cardiac Sparing

Growing evidence suggests that radiation therapy (RT) doses to the heart and cardiac substructures (CS) are strongly linked to cardiac toxicities, though only the heart is considered clinically. This work aimed to utilize the superior soft-tissue contrast of magnetic resonance (MR) to segment CS, quantify uncertainties in their position, assess their effect on treatment planning and an MR-guided environment. Automatic substructure segmentation of 12 CS was completed using a novel hybrid MR/computed tomography (CT) atlas method and was improved upon using a 3-dimensional neural network (U-Net) from deep learning. Intra-fraction motion due to respiration was then quantified. The inter-fraction setup uncertainties utilizing a novel MR-linear accelerator were also quantified. Treatment planning comparisons were performed with and without substructure inclusions and methods to reduce radiation dose to sensitive CS were evaluated. Lastly, these described technologies (deep learning U-Net) were translated to an MR-linear accelerator and a segmentation pipeline was created. Automatic segmentations from the hybrid MR/CT atlas was able to generate accurate segmentations for the chambers and great vessels (Dice similarity coefficient (DSC) \u3e 0.75) but coronary artery segmentations were unsuccessful (DSC\u3c0.3). After implementing deep learning, DSC for the chambers and great vessels was ≥0.85 along with an improvement in the coronary arteries (DSC\u3e0.5). Similar accuracy was achieved when implementing deep learning for MR-guided RT. On average, automatic segmentations required ~10 minutes to generate per patient and deep learning only required 14 seconds. The inclusion of CS in the treatment planning process did not yield statistically significant changes in plan complexity, PTV, or OAR dose. Automatic segmentation results from deep learning pose major efficiency and accuracy gains for CS segmentation offering high potential for rapid implementation into radiation therapy planning for improved cardiac sparing. Introducing CS into RT planning for MR-guided RT presented an opportunity for more effective sparing with limited increase in plan complexity