A Survey on Deep Learning in Medical Image Registration: New Technologies, Uncertainty, Evaluation Metrics, and Beyond

Over the past decade, deep learning technologies have greatly advanced the field of medical image registration. The initial developments, such as ResNet-based and U-Net-based networks, laid the groundwork for deep learning-driven image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, and uncertainty estimation. These advancements have not only enriched the field of deformable image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration

A first step toward uncovering the truth about weight tuning in deformable image registration

Deformable image registration is currently predominantly solved by optimizing a weighted linear combination of objectives. Successfully tuning the weights associated with these objectives is not trivial, leading to trial-and-error approaches. Such an approach assumes an intuitive interplay between weights, optimization objectives, and target registration errors. However, it is not known whether this always holds for existing registration methods. To investigate the interplay between weights, optimization objectives, and registration errors, we employ multi-objective optimization. Here, objectives of interest are optimized simultaneously, causing a set of multiple optimal solutions to exist, called the optimal Pareto front. Our medical application is in breast cancer and includes the challenging prone-supine registration problem. In total, we studied the interplay in three different ways. First, we ran many random linear combinations of objectives using the well-known registration software elastix. Second, since the optimization algorithms used in registration are typically of a local-search nature, final solutions may not always form a Pareto front. We therefore employed a multi-objective evolutionary algorithm that finds weights that correspond to registration outcomes that do form a Pareto front. Third, we examined how the interplay differs if a true multi-objective (i.e., weight-free) image registration method is used. Results indicate that a trial-and-error weight-adaptation approach can be successful for the easy prone to prone breast image registration case, due to the absence of many local optima. With increasing problem difficulty the use of more advanced approaches can be of value in finding and selecting the optimal registration outcomes

Quantifying the reproducibility of lung ventilation images between 4-Dimensional Cone Beam CT and 4-Dimensional CT.

PURPOSE: Computed tomography ventilation imaging derived from four-dimensional cone beam CT (CTVI4DCBCT ) can complement existing 4DCT-based methods (CTVI4DCT ) to track lung function changes over a course of lung cancer radiation therapy. However, the accuracy of CTVI4DCBCT needs to be assessed since anatomic 4DCBCT has demonstrably poor image quality and small field of view (FOV) compared to treatment planning 4DCT. We perform a direct comparison between short interval CTVI4DCBCT and CTVI4DCT pairs to understand the patient specific image quality factors affecting the intermodality CTVI reproducibility in the clinic. METHODS AND MATERIALS: We analysed 51 pairs of 4DCBCT and 4DCT scans acquired within 1 day of each other for nine lung cancer patients. To assess the impact of image quality, CTVIs were derived from 4DCBCT scans reconstructed using both standard Feldkamp-Davis-Kress backprojection (CTVIFDK4DCBCT) and an iterative McKinnon-Bates Simultaneous Algebraic Reconstruction Technique (CTVIMKBSART4DCBCT). Also, the influence of FOV was assessed by deriving CTVIs from 4DCT scans that were cropped to a similar FOV as the 4DCBCT scans (CTVIcrop4DCT), or uncropped (CTVIuncrop4DCT). All CTVIs were derived by performing deformable image registration (DIR) between the exhale and inhale phases and evaluating the Jacobian determinant of deformation. Reproducibility between corresponding CTVI4DCBCT and CTVI4DCT pairs was quantified using the voxel-wise Spearman rank correlation and the Dice similarity coefficient (DSC) for ventilation defect regions (identified as the lower quartile of ventilation values). Mann-Whitney U-tests were applied to determine statistical significance of each reconstruction and cropping condition. RESULTS: The (mean ± SD) Spearman correlation between CTVIFDK4DCBCT and CTVIuncrop4DCT was 0.60 ± 0.23 (range -0.03-0.88) and the DSC was 0.64 ± 0.12 (0.34-0.83). By comparison, correlations between CTVIMKBSART4DCBCT and CTVIuncrop4DCT showed a small but statistically significant improvement with = 0.64 ± 0.20 (range 0.06-0.90, P = 0.03) and DSC = 0.66 ± 0.13 (0.31-0.87, P = 0.02). Intermodal correlations were noted to decrease with an increasing fraction of lung truncation in 4DCBCT relative to 4DCT, albeit not significantly (Pearson correlation R = 0.58, P = 0.002). CONCLUSIONS: This study demonstrates that DIR based CTVIs derived from 4DCBCT can exhibit reasonable to good voxel-level agreement with CTVIs derived from 4DCT. These correlations outperform previous cross-modality comparisons between 4DCT-based ventilation and nuclear medicine. The use of 4DCBCT scans with iterative reconstruction and minimal lung truncation is recommended to ensure better reproducibility between 4DCBCT- and 4DCT-based CTVIs

Probabilistic non-linear registration with spatially adaptive regularisation

This paper introduces a novel method for inferring spatially varying regularisation in non-linear registration. This is achieved through full Bayesian inference on a probabilistic registration model, where the prior on the transformation parameters is parameterised as a weighted mixture of spatially localised components. Such an approach has the advantage of allowing the registration to be more flexibly driven by the data than a traditional globally defined regularisation penalty, such as bending energy. The proposed method adaptively determines the influence of the prior in a local region. The strength of the prior may be reduced in areas where the data better support deformations, or can enforce a stronger constraint in less informative areas. Consequently, the use of such a spatially adaptive prior may reduce unwanted impacts of regularisation on the inferred transformation. This is especially important for applications where the deformation field itself is of interest, such as tensor based morphometry. The proposed approach is demonstrated using synthetic images, and with application to tensor based morphometry analysis of subjects with Alzheimer’s disease and healthy controls. The results indicate that using the proposed spatially adaptive prior leads to sparser deformations, which provide better localisation of regional volume change. Additionally, the proposed regularisation model leads to more data driven and localised maps of registration uncertainty. This paper also demonstrates for the first time the use of Bayesian model comparison for selecting different types of regularisation

Computerized Analysis of Magnetic Resonance Images to Study Cerebral Anatomy in Developing Neonates

The study of cerebral anatomy in developing neonates is of great importance for the understanding of brain development during the early period of life. This dissertation therefore focuses on three challenges in the modelling of cerebral anatomy in neonates during brain development. The methods that have been developed all use Magnetic Resonance Images (MRI) as source data. To facilitate study of vascular development in the neonatal period, a set of image analysis algorithms are developed to automatically extract and model cerebral vessel trees. The whole process consists of cerebral vessel tracking from automatically placed seed points, vessel tree generation, and vasculature registration and matching. These algorithms have been tested on clinical Time-of- Flight (TOF) MR angiographic datasets. To facilitate study of the neonatal cortex a complete cerebral cortex segmentation and reconstruction pipeline has been developed. Segmentation of the neonatal cortex is not effectively done by existing algorithms designed for the adult brain because the contrast between grey and white matter is reversed. This causes pixels containing tissue mixtures to be incorrectly labelled by conventional methods. The neonatal cortical segmentation method that has been developed is based on a novel expectation-maximization (EM) method with explicit correction for mislabelled partial volume voxels. Based on the resulting cortical segmentation, an implicit surface evolution technique is adopted for the reconstruction of the cortex in neonates. The performance of the method is investigated by performing a detailed landmark study. To facilitate study of cortical development, a cortical surface registration algorithm for aligning the cortical surface is developed. The method first inflates extracted cortical surfaces and then performs a non-rigid surface registration using free-form deformations (FFDs) to remove residual alignment. Validation experiments using data labelled by an expert observer demonstrate that the method can capture local changes and follow the growth of specific sulcus

A Locally Adaptive Regularization Based on Anisotropic Diffusion for Deformable Image Registration of Sliding Organs

We propose a deformable image registration algorithm that uses anisotropic smoothing for regularization to find correspondences between images of sliding organs. In particular, we apply the method for respiratory motion estimation in longitudinal thoracic and abdominal computed tomography scans. The algorithm uses locally adaptive diffusion tensors to determine the direction and magnitude with which to smooth the components of the displacement field that are normal and tangential to an expected sliding boundary. Validation was performed using synthetic, phantom, and 14 clinical datasets, including the publicly available DIR-Lab dataset. We show that motion discontinuities caused by sliding can be effectively recovered, unlike conventional regularizations that enforce globally smooth motion. In the clinical datasets, target registration error showed improved accuracy for lung landmarks compared to the diffusive regularization. We also present a generalization of our algorithm to other sliding geometries, including sliding tubes (e.g., needles sliding through tissue, or contrast agent flowing through a vessel). Potential clinical applications of this method include longitudinal change detection and radiotherapy for lung or abdominal tumours, especially those near the chest or abdominal wall