A multi-resolution strategy for a multi-objective deformable image registration framework that accommodates large anatomical differences

Currently, two major challenges dominate the field of deformable image registration. The first challenge is related to the tuning of the developed methods to specific problems (i.e. how to best combine different objectives such as similarity measure and transformation effort). This is one of the reasons why, despite significant progress, clinical implementation of such techniques has proven to be difficult. The second challenge is to account for large anatomical differences (e.g. large deformations, (dis)appearing structures) that occurred between image acquisitions. In this paper, we study a framework based on multi-objective optimization to improve registration robustness and to simplify tuning for specific applications. Within this framework we specifically consider the use of an advanced model-based evolutionary algorithm for optimization and a dual-dynamic transformation model (i.e. two "non-fixed" grids: one for the source- and one for the target image) to accommodate for large anatomical differences. The framework computes and presents multiple outcomes that represent efficient trade-offs between the different objectives (a so-called Pareto front). In image processing it is common practice, for reasons of robustness and accuracy, to use a multi-resolution strategy. This is, however, only well-established for single-objective registration methods. Here we describe how such a strategy can be realized for our multi-objective approach and compare its results with a single-resolution strategy. For this study we selected the case of prone-supine breast MRI registration. Results show that the well-known advantages of a multi-resolution strategy are successfully transferred to our multi-objective approach, resulting in superior (i.e. Pareto-dominating) outcomes

Smart grid initialization reduces the computational complexity of multi-objective image registration based on a dual-dynamic transformation model to account for large anatomical differences

We recently demonstrated the strong potential of using dual-dynamic transformation models when tackling deformable image registration problems involving large anatomical differences. Dual-dynamic transformation models employ two moving grids instead of the common single moving grid for the target image (and single fixed grid for the source image). We previously employed powerful optimization algorithms to make use of the additional flexibility offered by a dual-dynamic transformation model with good results, directly obtaining insight into the trade-off between important registration objectives as a result of taking a multi-objective approach to optimization. However, optimization has so far been initialized using two regular grids, which still leaves a great potential of dual-dynamic transformation models untapped: a-priori grid alignment with image structures/areas that are expected to deform more. This allows (far) less grid points to be used, compared to using a sufficiently refined regular grid, leading to (far) more efficient optimization, or, equivalently, more accurate results using the same number of grid points. We study the implications of exploiting this potential by experimenting with two new smart grid initialization procedures: one manual expert-based and one automated image-feature-based. We consider a CT test case with large differences in bladder volume with and without a multi-resolution scheme and find a substantial benefit of using smart grid initialization

Getting the most out of additional guidance information in deformable image registration by leveraging multi-objective optimization

Incorporating additional guidance information, e.g., landmark/contour correspondence, in deformable image registration is often desirable and is typically done by adding constraints or cost terms to the optimization function. Commonly, deciding between a “hard” constraint and a “soft” additional cost term as well as the weighting of cost terms in the optimization function is done on a trial-and-error basis. The aim of this study is to investigate the advantages of exploiting guidance information by taking a multi-objective optimization perspective. Hereto, next to objectives related to match quality and amount of deformation, we define a third objective related to guidance information. Multi-objective optimization eliminates the need to a-priori tune a weighting of objectives in a single optimization function or the strict requirement of fulfilling hard guidance constraints. Instead, Pareto-efficient trade-offs between all objectives are found, effectively making the introduction of guidance information straightforward, independent of its type or scale. Further, since complete Pareto fronts also contain less interesting parts (i.e., solutions with near-zero deformation effort), we study how adaptive steering mechanisms can be incorporated to automatically focus more on solutions of interest. We performed experiments on artificial and real clinical data with large differences, including disappearing structures. Results show the substantial benefit of using additional guidance information. Moreover, compared to the 2-objective case, additional computational cost is negligible. Finally, with the same computational budget, use of the adaptive steering mechanism provides superior solutions in the area of interest

A novel model-based evolutionary algorithm for multi-objective deformable image registration with content mismatch and large deformations: Benchmarking efficiency and quality

Taking a multi-objective optimization approach to deformable image registration has recently gained attention, because such an approach removes the requirement of manually tuning the weights of all the involved objectives. Especially for problems that require large complex deformations, this is a non-trivial task. From the resulting Pareto set of solutions one can then much more insightfully select a registration outcome that is most suitable for the problem at hand. To serve as an internal optimization engine, currently used multi-objective algorithms are competent, but rather inefficient. In this paper we largely improve upon this by introducing a multi-objective real-valued adaptation of the recently introduced Gene-pool Optimal Mixing Evolutionary Algorithm (GOMEA) for discrete optimization. In this work, GOMEA is tailored specifically to the problem of deformable image registration to obtain substantially improved efficiency. This improvement is achieved by exploiting a key strength of GOMEA: iteratively improving small parts of solutions, allowing to faster exploit the impact of such updates on the objectives at hand through partial evaluations. We performed experiments on three registration problems. In particular, an artificial problem containing a disappearing structure, a pair of pre- and post-operative breast CT scans, and a pair of breast MRI scans acquired in prone and supine position were considered. Results show that compared to the previously used evolutionary algorithm, GOMEA obtains a speed-up of up to a factor of ∼1600 on the tested registration problems while achieving registration outcomes of similar quality

On the usefulness of gradient information in multi-objective deformable image registration using a B-spline-based dual-dynamic transformation model: comparison of three optimization algorithms

The use of gradient information is well-known to be highly useful in single-objective optimization-based image registration methods. However, its usefulness has not yet been investigated for deformable image registration from a multi-objective optimization perspective. To this end, within a previously introduced multi-objective optimization framework, we use a smooth B-spline-based dual-dynamic transformation model that allows us to derive gradient information analytically, while still being able to account for large deformations. Within the multi-objective framework, we previously employed a powerful evolutionary algorithm (EA) that computes and advances multiple outcomes at once, resulting in a set of solutions (a so-called Pareto front) that represents efficient trade-offs between the objectives. With the addition of the B-spline-based transformation model, we studied the usefulness of gradient information in multiobjective deformable image registration using three different optimization algorithms: the (gradient-less) EA, a gradientonly algorithm, and a hybridization of these two. We evaluated the algorithms to register highly deformed images: 2D MRI slices of the breast in prone and supine positions. Results demonstrate that gradient-based multi-objective optimization significantly speeds up optimization in the initial stages of optimization. However, allowing sufficient computational resources, better results could still be obtained with the EA. Ultimately, the hybrid EA found the best overall approximation of the optimal Pareto front, further indicating that adding gradient-based optimization for multiobjective optimization-based deformable image registration can indeed be beneficial

A first step toward uncovering the truth about weight tuning in deformable image registration

Deformable image registration is currently predominantly solved by optimizing a weighted linear combination of objectives. Successfully tuning the weights associated with these objectives is not trivial, leading to trial-and-error approaches. Such an approach assumes an intuitive interplay between weights, optimization objectives, and target registration errors. However, it is not known whether this always holds for existing registration methods. To investigate the interplay between weights, optimization objectives, and registration errors, we employ multi-objective optimization. Here, objectives of interest are optimized simultaneously, causing a set of multiple optimal solutions to exist, called the optimal Pareto front. Our medical application is in breast cancer and includes the challenging prone-supine registration problem. In total, we studied the interplay in three different ways. First, we ran many random linear combinations of objectives using the well-known registration software elastix. Second, since the optimization algorithms used in registration are typically of a local-search nature, final solutions may not always form a Pareto front. We therefore employed a multi-objective evolutionary algorithm that finds weights that correspond to registration outcomes that do form a Pareto front. Third, we examined how the interplay differs if a true multi-objective (i.e., weight-free) image registration method is used. Results indicate that a trial-and-error weight-adaptation approach can be successful for the easy prone to prone breast image registration case, due to the absence of many local optima. With increasing problem difficulty the use of more advanced approaches can be of value in finding and selecting the optimal registration outcomes

Multi-objective dual simplex-mesh based deformable image registration for 3D medical images – proof of concept

Reliably and physically accurately transferring information between images through deformable image registration with large anatomical differences is an open challenge in medical image analysis. Most existing methods have two key shortcomings: first, they require extensive up-front parameter tuning to each specific registration problem, and second, they have difficulty capturing large deformations and content mismatches between images. There have however been developments that have laid the foundation for potential solutions to both shortcomings. Towards the first shortcoming, a multi-objective optimization approach using the Real-Valued Gene-pool Optimal Mixing Evolutionary Algorithm (RV-GOMEA) has been shown to be capable of producing a diverse set of registrations for 2D images in one run of the algorithm, representing different trade-offs between conflicting objectives in the registration problem. This allows the user to select a registration afterwards and removes the need for up-front tuning. Towards the second shortcoming, a dual-dynamic grid transformation model has proven effective at capturing large differences in 2D images. These two developments have recently been accelerated through GPU parallelization, delivering large speed-ups. Based on this accelerated version, it is now possible to extend the approach to 3D images. Concordantly, this work introduces the first method for multi-objective 3D deformable image registration, using a 3D dual-dynamic grid transformation model based on simplex meshes while still supporting the incorporation of annotated guidance information and multi-resolution schemes. Our proof-of-concept prototype shows promising results on synthetic and clinical 3D registration problems, forming the foundation for a new, insightful method that can include bio-mechanical properties in the registration

Diversifying Multi-Objective Gradient Techniques and their Role in Hybrid Multi-Objective Evolutionary Algorithms for Deformable Medical Image Registration

Gradient methods and their value in single-objective, real-valued optimization are well-established. As such, they play a key role in tackling real-world, hard optimization problems such as deformable image registration (DIR). A key question is to which extent gradient techniques can also play a role in a multi-objective approach to DIR. We therefore aim to exploit gradient information within an evolutionary-algorithm-based multi-objective optimization framework for DIR. Although an analytical description of the multi-objective gradient (the set of all Pareto-optimal improving directions) is available, it is nontrivial how to best choose the most appropriate direction per solution because these directions are not necessarily uniformly distributed in objective space. To address this, we employ a Monte-Carlo method to obtain a discrete, spatially-uniformly distributed approximation of the set of Pareto-optimal improving directions. We then apply a diversification technique in which each solution is associated with a unique direction from this set based on its multi- as well as single-objective rank. To assess its utility, we compare a state-of-the-art multi-objective evolutionary algorithm with three different hybrid versions thereof on several benchmark problems and two medical DIR problems. Results show that the diversification strategy successfully leads to unbiased improvement, helping an adaptive hybrid scheme solve all problems, but the evolutionary algorithm remains the most powerful optimization method, providing the best balance between proximity and diversity

A Survey on Deep Learning in Medical Image Registration: New Technologies, Uncertainty, Evaluation Metrics, and Beyond

Over the past decade, deep learning technologies have greatly advanced the field of medical image registration. The initial developments, such as ResNet-based and U-Net-based networks, laid the groundwork for deep learning-driven image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, and uncertainty estimation. These advancements have not only enriched the field of deformable image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration