Artificial General Intelligence for Radiation Oncology

The emergence of artificial general intelligence (AGI) is transforming radiation oncology. As prominent vanguards of AGI, large language models (LLMs) such as GPT-4 and PaLM 2 can process extensive texts and large vision models (LVMs) such as the Segment Anything Model (SAM) can process extensive imaging data to enhance the efficiency and precision of radiation therapy. This paper explores full-spectrum applications of AGI across radiation oncology including initial consultation, simulation, treatment planning, treatment delivery, treatment verification, and patient follow-up. The fusion of vision data with LLMs also creates powerful multimodal models that elucidate nuanced clinical patterns. Together, AGI promises to catalyze a shift towards data-driven, personalized radiation therapy. However, these models should complement human expertise and care. This paper provides an overview of how AGI can transform radiation oncology to elevate the standard of patient care in radiation oncology, with the key insight being AGI's ability to exploit multimodal clinical data at scale

Deep learning-based affine and deformable 3D medical image registration

In medical image registration, medical scans are transformed to align their image content. Traditionally, image registration is performed manually by clinicians or using optimization-based algorithms, but in the past few years, deep learning has been successfully applied to the problem. In this work, deep learning image registration (DLIR) methods were compared on the task of aligning inter- and intra-patient male pelvic full field-of-view 3D Computed Tomography (CT) scans. The multistage registration pipeline used consisted of a cascade of an affine (global) registration and a deformable (local) registration. For the affine registration step, a 3D ResNet model was used. The two deformable methods that were investigated are VoxelMorph, the most commonly used DLIR framework, and LapIRN, a recent multi-resolution DLIR method. The two registration steps were trained separately; For the affine registration step, both supervised and unsupervised learning methods were employed. For the deformable step, unsupervised learning and weakly supervised learning using masks of regions of interest (ROIs) were used. The training was done on synthetically augmented CT scans. The results were compared to results obtained with two top-performing iterative image registration frameworks. The evaluation was based on ROI similarity of the registered scans, as well as diffeomorphic properties and runtime of the registration. Overall, the DLIR methods were not able to outperform the baseline iterative methods. The affine step followed by deformable registration with LaPIRN managed to perform similarly to or slightly worse than the baseline methods, managing to outperform them on 7 out of 12 ROIs on the intra-patient scans. The inter-patient registration task turned out to be challenging, with none of the methods performing well consistently. For both tasks, the DLIR methods achieve a very significant time speedup compared to the baseline methods