Registration and analysis of dynamic magnetic resonance image series

Cystic fibrosis (CF) is an autosomal-recessive inherited metabolic disorder that affects all organs in the human body. Patients affected with CF suffer particularly from chronic inflammation and obstruction of the airways. Through early detection, continuous monitoring methods, and new treatments, the life expectancy of patients with CF has been increased drastically in the last decades. However, continuous monitoring of the disease progression is essential for a successful treatment. The current state-of-the-art method for lung disease detection and monitoring is computed tomography (CT) or X-ray. These techniques are ill-suited for the monitoring of disease progressions because of the ionizing radiation the patient is exposed during the examination. Through the development of new magnetic resonance imaging (MRI) sequences and evaluation methods, MRI is able to measure physiological changes in the lungs. The process to create physiological maps, i.e. ventilation and perfusion maps, of the lungs using MRI can be split up into three parts: MR-acquisition, image registration, and image analysis. In this work, we present different methods for the image registration part and the image analysis part. We developed a graph-based registration method for 2D dynamic MR image series of the lungs in order to overcome the problem of sliding motion at organ boundaries. Furthermore, we developed a human-inspired learning-based registration method. Here, the registration is defined as a sequence of local transformations. The sequence-based approach combines the advantage of dense transformation models, i.e. large space of transformations, and the advantage of interpolating transformation models, i.e. smooth local transformations. We also developed a general registration framework called Autograd Image Registration Laboratory (AIRLab), which performs automatic calculation of the gradients for the registration process. This allows rapid prototyping and an easy implementation of existing registration algorithms. For the image analysis part, we developed a deep-learning approach based on gated recurrent units that are able to calculate ventilation maps with less than a third of the number of images of the current method. Automatic defect detection in the estimated MRI ventilation and perfusion maps is essential for the clinical routine to automatically evaluate the treatment progression. We developed a weakly supervised method that is able to infer a pixel-wise defect segmentation by using only a continuous global label during training. In this case, we directly use the lung clearance index (LCI) as a global weak label, without any further manual annotations. The LCI is a global measure to describe ventilation inhomogeneities of the lungs and is obtained by a multiple breath washout test

Joint segmentation and discontinuity-preserving deformable registration: Application to cardiac cine-MR images

Medical image registration is a challenging task involving the estimation of spatial transformations to establish anatomical correspondence between pairs or groups of images. Recently, deep learning-based image registration methods have been widely explored, and demonstrated to enable fast and accurate image registration in a variety of applications. However, most deep learning-based registration methods assume that the deformation fields are smooth and continuous everywhere in the image domain, which is not always true, especially when registering images whose fields of view contain discontinuities at tissue/organ boundaries. In such scenarios, enforcing smooth, globally continuous deformation fields leads to incorrect/implausible registration results. We propose a novel discontinuity-preserving image registration method to tackle this challenge, which ensures globally discontinuous and locally smooth deformation fields, leading to more accurate and realistic registration results. The proposed method leverages the complementary nature of image segmentation and registration and enables joint segmentation and pair-wise registration of images. A co-attention block is proposed in the segmentation component of the network to learn the structural correlations in the input images, while a discontinuity-preserving registration strategy is employed in the registration component of the network to ensure plausibility in the estimated deformation fields at tissue/organ interfaces. We evaluate our method on the task of intra-subject spatio-temporal image registration using large-scale cinematic cardiac magnetic resonance image sequences, and demonstrate that our method achieves significant improvements over the state-of-the-art for medical image registration, and produces high-quality segmentation masks for the regions of interest

Non-rigid medical image registration with extended free form deformations: modelling general tissue transitions

Image registration seeks pointwise correspondences between the same or analogous objects in different images. Conventional registration methods generally impose continuity and smoothness throughout the image. However, there are cases in which the deformations may involve discontinuities. In general, the discontinuities can be of different types, depending on the physical properties of the tissue transitions involved and boundary conditions. For instance, in the respiratory motion the lungs slide along the thoracic cage following the tangential direction of their interface. In the normal direction, however, the lungs and the thoracic cage are constrained to be always in contact but they have different material properties producing different compression or expansion rates. In the literature, there is no generic method, which handles different types of discontinuities and considers their directional dependence. The aim of this thesis is to develop a general registration framework that is able to correctly model different types of tissue transitions with a general formalism. This has led to the development of the eXtended Free Form Deformation (XFFD) registration method. XFFD borrows the concept of the interpolation method from the eXtended Finite Element method (XFEM) to incorporate discontinuities by enriching B-spline basis functions, coupled with extra degrees of freedom. XFFD can handle different types of discontinuities and encodes their directional-dependence without any additional constraints. XFFD has been evaluated on digital phantoms, publicly available 3D liver and lung CT images. The experiments show that XFFD improves on previous methods and that it is important to employ the correct model that corresponds to the discontinuity type involved at the tissue transition. The effect of using incorrect models is more evident in the strain, which measures mechanical properties of the tissues

Optical flow estimation using steered-L1 norm

Motion is a very important part of understanding the visual picture of the surrounding environment. In image processing it involves the estimation of displacements for image points in an image sequence. In this context dense optical flow estimation is concerned with the computation of pixel displacements in a sequence of images, therefore it has been used widely in the field of image processing and computer vision. A lot of research was dedicated to enable an accurate and fast motion computation in image sequences. Despite the recent advances in the computation of optical flow, there is still room for improvements and optical flow algorithms still suffer from several issues, such as motion discontinuities, occlusion handling, and robustness to illumination changes. This thesis includes an investigation for the topic of optical flow and its applications. It addresses several issues in the computation of dense optical flow and proposes solutions. Specifically, this thesis is divided into two main parts dedicated to address two main areas of interest in optical flow. In the first part, image registration using optical flow is investigated. Both local and global image registration has been used for image registration. An image registration based on an improved version of the combined Local-global method of optical flow computation is proposed. A bi-lateral filter was used in this optical flow method to improve the edge preserving performance. It is shown that image registration via this method gives more robust results compared to the local and the global optical flow methods previously investigated. The second part of this thesis encompasses the main contribution of this research which is an improved total variation L1 norm. A smoothness term is used in the optical flow energy function to regularise this function. The L1 is a plausible choice for such a term because of its performance in preserving edges, however this term is known to be isotropic and hence decreases the penalisation near motion boundaries in all directions. The proposed improved L1 (termed here as the steered-L1 norm) smoothness term demonstrates similar performance across motion boundaries but improves the penalisation performance along such boundaries

Dense Vision in Image-guided Surgery

Image-guided surgery needs an efficient and effective camera tracking system in order to perform augmented reality for overlaying preoperative models or label cancerous tissues on the 2D video images of the surgical scene. Tracking in endoscopic/laparoscopic scenes however is an extremely difficult task primarily due to tissue deformation, instrument invasion into the surgical scene and the presence of specular highlights. State of the art feature-based SLAM systems such as PTAM fail in tracking such scenes since the number of good features to track is very limited. When the scene is smoky and when there are instrument motions, it will cause feature-based tracking to fail immediately. The work of this thesis provides a systematic approach to this problem using dense vision. We initially attempted to register a 3D preoperative model with multiple 2D endoscopic/laparoscopic images using a dense method but this approach did not perform well. We subsequently proposed stereo reconstruction to directly obtain the 3D structure of the scene. By using the dense reconstructed model together with robust estimation, we demonstrate that dense stereo tracking can be incredibly robust even within extremely challenging endoscopic/laparoscopic scenes. Several validation experiments have been conducted in this thesis. The proposed stereo reconstruction algorithm has turned out to be the state of the art method for several publicly available ground truth datasets. Furthermore, the proposed robust dense stereo tracking algorithm has been proved highly accurate in synthetic environment (< 0.1 mm RMSE) and qualitatively extremely robust when being applied to real scenes in RALP prostatectomy surgery. This is an important step toward achieving accurate image-guided laparoscopic surgery.Open Acces

(An overview of) Synergistic reconstruction for multimodality/multichannel imaging methods

Imaging is omnipresent in modern society with imaging devices based on a zoo of physical principles, probing a specimen across different wavelengths, energies and time. Recent years have seen a change in the imaging landscape with more and more imaging devices combining that which previously was used separately. Motivated by these hardware developments, an ever increasing set of mathematical ideas is appearing regarding how data from different imaging modalities or channels can be synergistically combined in the image reconstruction process, exploiting structural and/or functional correlations between the multiple images. Here we review these developments, give pointers to important challenges and provide an outlook as to how the field may develop in the forthcoming years. This article is part of the theme issue 'Synergistic tomographic image reconstruction: part 1'

Multiresolution eXtended Free-Form Deformations (XFFD) for non-rigid registration with discontinuous transforms

Image registration is an essential technique to obtain point correspondences between anatomical structures from different images. Conventional non-rigid registration methods assume a continuous and smooth deformation field throughout the image. However, the deformation field at the interface of different organs is not necessarily continuous, since the organs may slide over or separate from each other. Therefore, imposing continuity and smoothness ubiquitously would lead to artifacts and increased errors near the discontinuity interface. In computational mechanics, the eXtended Finite Element Method (XFEM) was introduced to handle discontinuities without using computational meshes that conform to the discontinuity geometry. Instead, the interpolation bases themselves were enriched with discontinuous functional terms. Borrowing this concept, we propose a multiresolution eXtented Free-Form Deformation (XFFD) framework that seamlessly integrates within and extends the standard Free-Form Deformation (FFD) approach. Discontinuities are incorporated by enriching the B-spline basis functions coupled with extra degrees of freedom, which are only introduced near the discontinuity interface. In contrast with most previous methods, restricted to sliding motion, no ad hoc penalties or constraints are introduced to reduce gaps and overlaps. This allows XFFD to describe more general discontinuous motions. In addition, we integrate XFFD into a rigorously formulated multiresolution framework by introducing an exact parameter upsampling method. The proposed method has been evaluated in two publicly available datasets: 4D pulmonary CT images from the DIR-Lab dataset and 4D CT liver datasets. The XFFD achieved a Target Registration Error (TRE) of 1.17 ± 0.85 mm in the DIR-lab dataset and 1.94 ± 1.01 mm in the liver dataset, which significantly improves on the performance of the state-of-the-art methods handling discontinuities

Correspondence problems in computer vision : novel models, numerics, and applications

Correspondence problems like optic flow belong to the fundamental problems in computer vision. Here, one aims at finding correspondences between the pixels in two (or more) images. The correspondences are described by a displacement vector field that is often found by minimising an energy (cost) function. In this thesis, we present several contributions to the energy-based solution of correspondence problems: (i) We start by developing a robust data term with a high degree of invariance under illumination changes. Then, we design an anisotropic smoothness term that works complementary to the data term, thereby avoiding undesirable interference. Additionally, we propose a simple method for determining the optimal balance between the two terms. (ii) When discretising image derivatives that occur in our continuous models, we show that adapting one-sided upwind discretisations from the field of hyperbolic differential equations can be beneficial. To ensure a fast solution of the nonlinear system of equations that arises when minimising the energy, we use the recent fast explicit diffusion (FED) solver in an explicit gradient descent scheme. (iii) Finally, we present a novel application of modern optic flow methods where we align exposure series used in high dynamic range (HDR) imaging. Furthermore, we show how the alignment information can be used in a joint super-resolution and HDR method.Korrespondenzprobleme wie der optische Fluß, gehören zu den fundamentalen Problemen im Bereich des maschinellen Sehens (Computer Vision). Hierbei ist das Ziel, Korrespondenzen zwischen den Pixeln in zwei (oder mehreren) Bildern zu finden. Die Korrespondenzen werden durch ein Verschiebungsvektorfeld beschrieben, welches oft durch Minimierung einer Energiefunktion (Kostenfunktion) gefunden wird. In dieser Arbeit stellen wir mehrere Beiträge zur energiebasierten Lösung von Korrespondenzproblemen vor: (i) Wir beginnen mit der Entwicklung eines robusten Datenterms, der ein hohes Maß an Invarianz unter Beleuchtungsänderungen aufweißt. Danach entwickeln wir einen anisotropen Glattheitsterm, der komplementär zu dem Datenterm wirkt und deshalb keine unerwünschten Interferenzen erzeugt. Zusätzlich schlagen wir eine einfache Methode vor, die es erlaubt die optimale Balance zwischen den beiden Termen zu bestimmen. (ii) Im Zuge der Diskretisierung von Bildableitungen, die in unseren kontinuierlichen Modellen auftauchen, zeigen wir dass es hilfreich sein kann, einseitige upwind Diskretisierungen aus dem Bereich hyperbolischer Differentialgleichungen zu übernehmen. Um eine schnelle Lösung des nichtlinearen Gleichungssystems, dass bei der Minimierung der Energie auftaucht, zu gewährleisten, nutzen wir den kürzlich vorgestellten fast explicit diffusion (FED) Löser im Rahmen eines expliziten Gradientenabstiegsschemas. (iii) Schließlich stellen wir eine neue Anwendung von modernen optischen Flußmethoden vor, bei der Belichtungsreihen für high dynamic range (HDR) Bildgebung registriert werden. Außerdem zeigen wir, wie diese Registrierungsinformation in einer kombinierten super-resolution und HDR Methode genutzt werden kann