Methods for three-dimensional Registration of Multimodal Abdominal Image Data

Multimodal image registration benefits the diagnosis, treatment planning and the performance of image-guided procedures in the liver, since it enables the fusion of complementary information provided by pre- and intrainterventional data about tumor localization and access. Although there exist various registration methods, approaches which are specifically optimized for the registration of multimodal abdominal scans are only scarcely available. The work presented in this thesis aims to tackle this problem by focusing on the development, optimization and evaluation of registration methods specifically for the registration of multimodal liver scans. The contributions to the research field of medical image registration include the development of a registration evaluation methodology that enables the comparison and optimization of linear and non-linear registration algorithms using a point-based accuracy measure. This methodology has been used to benchmark standard registration methods as well as novel approaches that were developed within the frame of this thesis. The results of the methodology showed that the employed similarity measure used during the registration has a major impact on the registration accuracy of the method. Due to this influence, two alternative similarity metrics bearing the potential to be used on multimodal image data are proposed and evaluated. The first metric relies on the use of gradient information in form of Histograms of Oriented Gradients (HOG) whereas the second metric employs a siamese neural network to learn a similarity measure directly on the image data. The evaluation showed, that both metrics could compete with state of the art similarity measures in terms of registration accuracy. The HOG-metric offers the advantage that it does not require ground truth data to learn a similarity estimation, but instead it is applicable to various data sets with the sole requirement of distinct gradients. However, the Siamese metric is characterized by a higher robustness for large rotations than the HOG-metric. To train such a network, registered ground truth data is required which may be critical for multimodal image data. Yet, the results show that it is possible to apply models trained on registered synthetic data on real patient data. The last part of this thesis focuses on methods to learn an entire registration process using neural networks, thereby offering the advantage to replace the traditional, time-consuming iterative registration procedure. Within the frame of this thesis, the so-called VoxelMorph network which was originally proposed for monomodal, non-linear registration learning is extended for affine and multimodal registration learning tasks. This extension includes the consideration of an image mask during metric evaluation as well as loss functions for multimodal data, such as the pretrained Siamese metric and a loss relying on the comparison of deformation fields. Based on the developed registration evaluation methodology, the performance of the original network as well as the extended variants are evaluated for monomodal and multimodal registration tasks using multiple data sets. With the extended network variants, it is possible to learn an entire multimodal registration process for the correction of large image displacements. As for the Siamese metric, the results imply a general transferability of models trained with synthetic data to registration tasks including real patient data. Due to the lack of multimodal ground truth data, this transfer represents an important step towards making Deep Learning based registration procedures clinically usable

Automatic PET-CT Image Registration Method Based on Mutual Information and Genetic Algorithms

Hybrid PET/CT scanners can simultaneously visualize coronary artery disease as revealed by computed tomography (CT) and myocardial perfusion as measured by positron emission tomography (PET). Manual registration is usually required in clinical practice to compensate spatial mismatch between datasets. In this paper, we present a registration algorithm that is able to automatically align PET/CT cardiac images. The algorithm bases on mutual information (MI) as registration metric and on genetic algorithm as optimization method. A multiresolution approach was used to optimize the processing time. The algorithm was tested on computerized models of volumetric PET/CT cardiac data and on real PET/CT datasets. The proposed automatic registration algorithm smoothes the pattern of the MI and allows it to reach the global maximum of the similarity function. The implemented method also allows the definition of the correct spatial transformation that matches both synthetic and real PET and CT volumetric datasets

Intensity-based image registration using multiple distributed agents

Image registration is the process of geometrically aligning images taken from different sensors, viewpoints or instances in time. It plays a key role in the detection of defects or anomalies for automated visual inspection. A multiagent distributed blackboard system has been developed for intensity-based image registration. The images are divided into segments and allocated to agents on separate processors, allowing parallel computation of a similarity metric that measures the degree of likeness between reference and sensed images after the application of a transform. The need for a dedicated control module is removed by coordination of agents via the blackboard. Tests show that additional agents increase speed, provided the communication capacity of the blackboard is not saturated. The success of the approach in achieving registration, despite significant misalignment of the original images, is demonstrated in the detection of manufacturing defects on screen-printed plastic bottles and printed circuit boards

Geodesic Information Flows: Spatially-Variant Graphs and Their Application to Segmentation and Fusion

Clinical annotations, such as voxel-wise binary or probabilistic tissue segmentations, structural parcellations, pathological regionsof- interest and anatomical landmarks are key to many clinical studies. However, due to the time consuming nature of manually generating these annotations, they tend to be scarce and limited to small subsets of data. This work explores a novel framework to propagate voxel-wise annotations between morphologically dissimilar images by diffusing and mapping the available examples through intermediate steps. A spatially-variant graph structure connecting morphologically similar subjects is introduced over a database of images, enabling the gradual diffusion of information to all the subjects, even in the presence of large-scale morphological variability. We illustrate the utility of the proposed framework on two example applications: brain parcellation using categorical labels and tissue segmentation using probabilistic features. The application of the proposed method to categorical label fusion showed highly statistically significant improvements when compared to state-of-the-art methodologies. Significant improvements were also observed when applying the proposed framework to probabilistic tissue segmentation of both synthetic and real data, mainly in the presence of large morphological variability

Robust similarity metrics for the registration of 3D multimodal medical images

In this paper, we develop data driven registration algorithms, relying on pixel similarity metrics, that enable an accurate rigid registration of dissimilar single or multimodal 2D/3D medical images . Gross dissimilarities are handled by considering similarity measures related to robust M-estimators . Fast stochastic multigrid optimization algorithms are used to minimize these similarity metrics . The proposed robust similarity metrics are compared to the most popular standard similarity metrics on real MRI/MRI and MRI/SPECT image pairs showing gross dissimilarities . A blinded evaluation of the algorithm was performed, using as gold standard a prospective, marker-based registration method, by participating in a registration evaluation project (Vanderbilt University) . Our robust similarity measures compare favourably with all standard (non robust) techniques .Le recalage non supervisé d'images médicales volumiques reste un problème difficile en raison de l'importante variabilité et des grandes différences d'information pouvant apparaître dans des séquences d'images de même modalité ou dans des couples d'images multimodales. Nous présentons dans cet article des méthodes robustes de recalage rigide d'images 2D et 3D monomodales et multimodales, reposant sur la minimisation de mesures de similarité inter-images. Les méthodes proposées s'appuient sur la théorie de l'estimation robuste et mettent en oeuvre des M-estimateurs associés à des techniques d'optimisation stochastique multigrilles rapides. Ces estimateurs robustes sont évalués à travers le recalage d'images médicales volumiques monomodales (IRM/IRM) et multimodales (IRM/TEMP). Ils sont comparés aux autres fonctions de similarité classiques, proposées dans la littérature. Les méthodes de recalage robustes ont, en particulier, été validées dans le cadre d'un protocole comparatif mis en place par l'Université de Vanderbilt. Elles sont actuellement utilisées en routine clinique et conduisent, tant pour les images de même modalité que pour les images multimodales à une précision sous-voxel, comparable aux meilleures méthodes actuelles. Elles permettent de plus de recaler des couples d'images sur lesquels les méthodes classiques échouent

Synth-by-Reg (SbR): Contrastive Learning for Synthesis-Based Registration of Paired Images

Nonlinear inter-modality registration is often challenging due to the lack of objective functions that are good proxies for alignment. Here we propose a synthesis-by-registration method to convert this problem into an easier intra-modality task. We introduce a registration loss for weakly supervised image translation between domains that does not require perfectly aligned training data. This loss capitalises on a registration U-Net with frozen weights, to drive a synthesis CNN towards the desired translation. We complement this loss with a structure preserving constraint based on contrastive learning, which prevents blurring and content shifts due to overfitting. We apply this method to the registration of histological sections to MRI slices, a key step in 3D histology reconstruction. Results on two public datasets show improvements over registration based on mutual information (13% reduction in landmark error) and synthesis-based algorithms such as CycleGAN (11% reduction), and are comparable to registration with label supervision. Code and data are publicly available at https://github.com/acasamitjana/SynthByReg

Morphogenetic Principles of Brain Organisation in Health and Disease

Non-invasive neuroimaging methods, such as MRI, provide a window into the structure of the mammalian brain. However, despite the ubiquity of these methods, the biological interpretation of the information obtained using these tools remains elusive. In order to accurately link this macroscale data to microscale measurements, it is critical that the construct validity is high. This thesis provides novel analyses, pipelines and methods to: i) generate and validate maps of brain organisation obtained via MRI, and ii) demonstrate the utility of these methods in capturing elements of cognition and psychopathology. First, in Chapter 1, I review some of the neuroscientific context for the new methods presented, from cytoarchitecture to gene expression to connectomes. Chapters 2-4 introduce a new method, “Morphometric Similarity Mapping”, which captures the brain organisation of an individual by mapping the relationships of multiple features of the cerebral cortex. Chapter 2 focuses on the development of the analysis pipeline and the graph theoretical features of the resulting morphometric similarity networks (MSNs), with an emphasis on reproducibility. Chapter 3 highlights the generalisability of MSNs to the macaque monkey, linking MSNs to ex vivo tract tracing experiments and presenting new tools for processing non-human imaging data; as well as evidence that MSN topography is organised by cytoarchitectonic features. Chapter 4 is focused on determining the transcriptomic correlates of MSNs using publicly available gene expression maps, and on applying MSNs to examine the relationship between brain organisation and intelligence. Chapter 5 is dedicated to rigorous evaluation of the applicability of MSNs to measure specific disease-relevant phenotypes in 8 rare genetic disorder cohorts. This includes the validation of novel methods for utilising data from both single-cell sequencing technologies and differential gene expression experiments (in multiple tissue types) in analysing neuroimaging and bulk transcriptomic brain maps. Chapter 6 provides a brief summary and presents some ongoing and future projects expanding on this original work. It also importantly discusses a general framework of comparing brain maps, including MSNs and gene expression, as well as other canonical maps of brain structure and function. Altogether, this thesis presents and evaluates novel methods and applications for integrating multimodal neuroimaging data with genetic data derived from multiple tissue types and through various acquisition strategies. It also includes tools for performing these analyses in non-human primates, and pipelines for statistically comparing brain maps. These results not only provide insight into the manifestation of brain-related changes due to various components of human variation, but also provides a framework for evaluating this variation at multiple biological scales purely from non-invasive neuroimaging data

Combining Shape and Learning for Medical Image Analysis

Automatic methods with the ability to make accurate, fast and robust assessments of medical images are highly requested in medical research and clinical care. Excellent automatic algorithms are characterized by speed, allowing for scalability, and an accuracy comparable to an expert radiologist. They should produce morphologically and physiologically plausible results while generalizing well to unseen and rare anatomies. Still, there are few, if any, applications where today\u27s automatic methods succeed to meet these requirements.\ua0The focus of this thesis is two tasks essential for enabling automatic medical image assessment, medical image segmentation and medical image registration. Medical image registration, i.e. aligning two separate medical images, is used as an important sub-routine in many image analysis tools as well as in image fusion, disease progress tracking and population statistics. Medical image segmentation, i.e. delineating anatomically or physiologically meaningful boundaries, is used for both diagnostic and visualization purposes in a wide range of applications, e.g. in computer-aided diagnosis and surgery.The thesis comprises five papers addressing medical image registration and/or segmentation for a diverse set of applications and modalities, i.e. pericardium segmentation in cardiac CTA, brain region parcellation in MRI, multi-organ segmentation in CT, heart ventricle segmentation in cardiac ultrasound and tau PET registration. The five papers propose competitive registration and segmentation methods enabled by machine learning techniques, e.g. random decision forests and convolutional neural networks, as well as by shape modelling, e.g. multi-atlas segmentation and conditional random fields