Joint segmentation and discontinuity-preserving deformable registration: Application to cardiac cine-MR images

Medical image registration is a challenging task involving the estimation of spatial transformations to establish anatomical correspondence between pairs or groups of images. Recently, deep learning-based image registration methods have been widely explored, and demonstrated to enable fast and accurate image registration in a variety of applications. However, most deep learning-based registration methods assume that the deformation fields are smooth and continuous everywhere in the image domain, which is not always true, especially when registering images whose fields of view contain discontinuities at tissue/organ boundaries. In such scenarios, enforcing smooth, globally continuous deformation fields leads to incorrect/implausible registration results. We propose a novel discontinuity-preserving image registration method to tackle this challenge, which ensures globally discontinuous and locally smooth deformation fields, leading to more accurate and realistic registration results. The proposed method leverages the complementary nature of image segmentation and registration and enables joint segmentation and pair-wise registration of images. A co-attention block is proposed in the segmentation component of the network to learn the structural correlations in the input images, while a discontinuity-preserving registration strategy is employed in the registration component of the network to ensure plausibility in the estimated deformation fields at tissue/organ interfaces. We evaluate our method on the task of intra-subject spatio-temporal image registration using large-scale cinematic cardiac magnetic resonance image sequences, and demonstrate that our method achieves significant improvements over the state-of-the-art for medical image registration, and produces high-quality segmentation masks for the regions of interest

Registration and analysis of dynamic magnetic resonance image series

Cystic fibrosis (CF) is an autosomal-recessive inherited metabolic disorder that affects all organs in the human body. Patients affected with CF suffer particularly from chronic inflammation and obstruction of the airways. Through early detection, continuous monitoring methods, and new treatments, the life expectancy of patients with CF has been increased drastically in the last decades. However, continuous monitoring of the disease progression is essential for a successful treatment. The current state-of-the-art method for lung disease detection and monitoring is computed tomography (CT) or X-ray. These techniques are ill-suited for the monitoring of disease progressions because of the ionizing radiation the patient is exposed during the examination. Through the development of new magnetic resonance imaging (MRI) sequences and evaluation methods, MRI is able to measure physiological changes in the lungs. The process to create physiological maps, i.e. ventilation and perfusion maps, of the lungs using MRI can be split up into three parts: MR-acquisition, image registration, and image analysis. In this work, we present different methods for the image registration part and the image analysis part. We developed a graph-based registration method for 2D dynamic MR image series of the lungs in order to overcome the problem of sliding motion at organ boundaries. Furthermore, we developed a human-inspired learning-based registration method. Here, the registration is defined as a sequence of local transformations. The sequence-based approach combines the advantage of dense transformation models, i.e. large space of transformations, and the advantage of interpolating transformation models, i.e. smooth local transformations. We also developed a general registration framework called Autograd Image Registration Laboratory (AIRLab), which performs automatic calculation of the gradients for the registration process. This allows rapid prototyping and an easy implementation of existing registration algorithms. For the image analysis part, we developed a deep-learning approach based on gated recurrent units that are able to calculate ventilation maps with less than a third of the number of images of the current method. Automatic defect detection in the estimated MRI ventilation and perfusion maps is essential for the clinical routine to automatically evaluate the treatment progression. We developed a weakly supervised method that is able to infer a pixel-wise defect segmentation by using only a continuous global label during training. In this case, we directly use the lung clearance index (LCI) as a global weak label, without any further manual annotations. The LCI is a global measure to describe ventilation inhomogeneities of the lungs and is obtained by a multiple breath washout test

Segmentation and quantification of spinal cord gray matter–white matter structures in magnetic resonance images

This thesis focuses on finding ways to differentiate the gray matter (GM) and white matter (WM) in magnetic resonance (MR) images of the human spinal cord (SC). The aim of this project is to quantify tissue loss in these compartments to study their implications on the progression of multiple sclerosis (MS). To this end, we propose segmentation algorithms that we evaluated on MR images of healthy volunteers. Segmentation of GM and WM in MR images can be done manually by human experts, but manual segmentation is tedious and prone to intra- and inter-rater variability. Therefore, a deterministic automation of this task is necessary. On axial 2D images acquired with a recently proposed MR sequence, called AMIRA, we experiment with various automatic segmentation algorithms. We first use variational model-based segmentation approaches combined with appearance models and later directly apply supervised deep learning to train segmentation networks. Evaluation of the proposed methods shows accurate and precise results, which are on par with manual segmentations. We test the developed deep learning approach on images of conventional MR sequences in the context of a GM segmentation challenge, resulting in superior performance compared to the other competing methods. To further assess the quality of the AMIRA sequence, we apply an already published GM segmentation algorithm to our data, yielding higher accuracy than the same algorithm achieves on images of conventional MR sequences. On a different topic, but related to segmentation, we develop a high-order slice interpolation method to address the large slice distances of images acquired with the AMIRA protocol at different vertebral levels, enabling us to resample our data to intermediate slice positions. From the methodical point of view, this work provides an introduction to computer vision, a mathematically focused perspective on variational segmentation approaches and supervised deep learning, as well as a brief overview of the underlying project's anatomical and medical background

Variational Methods in Shape Space

This dissertation deals with the application of variational methods in spaces of geometric shapes. In particular, the treated topics include shape averaging, principal component analysis in shape space, computation of geodesic paths in shape space, as well as shape optimisation. Chapter 1 provides a brief overview over the employed models of shape space. Geometric shapes are identified with two- or three-dimensional, deformable objects. Deformations will be described via physical models; in particular, the objects will be interpreted as consisting of either a hyperelastic solid or a viscous liquid material. Furthermore, the description of shapes via phase fields or level sets is briefly introduced. Chapter 2 reviews different and related approaches to shape space modelling. References to related topics in image segmentation and registration are also provided. Finally, the relevant shape optimisation literature is introduced. Chapter 3 recapitulates the employed concepts from continuum mechanics and phase field modelling and states basic theoretical results needed for the later analysis. Chapter 4 addresses the computation of shape averages, based on a hyperelastic notion of shape dissimilarity: The dissimilarity between two shapes is measured as the minimum deformation energy required to deform the first into the second shape. A corresponding phase-field model is introduced, analysed, and finally implemented numerically via finite elements. A principal component analysis of shapes, which is consistent with the previously introduced average, is considered in Chapter 5. Elastic boundary stresses on the average shape are used as representatives of the input shapes in a linear vector space. On these linear representatives, a standard principal component analysis can be performed, where the employed covariance metric should be properly chosen to depend on the input shapes. Chapter 6 interprets shapes as belonging to objects made of a viscous liquid and correspondingly defines geodesic paths between shapes. The energy of a path is given as the total physical dissipation during the deformation of an object along the path. A rigid body motion invariant time discretisation is achieved by approximating the dissipation along a path segment by the deformation energy of a small solid deformation. The numerical implementation is based on level sets. Chapter 7 is concerned with the optimisation of the geometry and topology of solid structures that are subject to a mechanical load. Given the load configuration, the structure rigidity, its volume, and its surface area shall be optimally balanced. A phase field model is devised and analysed for this purpose. In this context, the use of nonlinear elasticity allows to detect buckling phenomena which would be ignored in linearised elasticity

Joint methods in imaging based on diffuse image representations

This thesis deals with the application and the analysis of different variants of the Mumford-Shah model in the context of image processing. In this kind of models, a given function is approximated in a piecewise smooth or piecewise constant manner. Especially the numerical treatment of the discontinuities requires additional models that are also outlined in this work. The main part of this thesis is concerned with four different topics. Simultaneous edge detection and registration of two images: The image edges are detected with the Ambrosio-Tortorelli model, an approximation of the Mumford-Shah model that approximates the discontinuity set with a phase field, and the registration is based on these edges. The registration obtained by this model is fully symmetric in the sense that the same matching is obtained if the roles of the two input images are swapped. Detection of grain boundaries from atomic scale images of metals or metal alloys: This is an image processing problem from materials science where atomic scale images are obtained either experimentally for instance by transmission electron microscopy or by numerical simulation tools. Grains are homogenous material regions whose atomic lattice orientation differs from their surroundings. Based on a Mumford-Shah type functional, the grain boundaries are modeled as the discontinuity set of the lattice orientation. In addition to the grain boundaries, the model incorporates the extraction of a global elastic deformation of the atomic lattice. Numerically, the discontinuity set is modeled by a level set function following the approach by Chan and Vese. Joint motion estimation and restoration of motion-blurred video: A variational model for joint object detection, motion estimation and deblurring of consecutive video frames is proposed. For this purpose, a new motion blur model is developed that accurately describes the blur also close to the boundary of a moving object. Here, the video is assumed to consist of an object moving in front of a static background. The segmentation into object and background is handled by a Mumford-Shah type aspect of the proposed model. Convexification of the binary Mumford-Shah segmentation model: After considering the application of Mumford-Shah type models to tackle specific image processing problems in the previous topics, the Mumford-Shah model itself is studied more closely. Inspired by the work of Nikolova, Esedoglu and Chan, a method is developed that allows global minimization of the binary Mumford-Shah segmentation model by solving a convex, unconstrained optimization problem. In an outlook, segmentation of flowfields into piecewise affine regions using this convexification method is briefly discussed

Reasoning about Geometric Object Interactions in 3D for Manipulation Action Understanding

In order to efficiently interact with human users, intelligent agents and autonomous systems need the ability of interpreting human actions. We focus our attention on manipulation actions, wherein an agent typically grasps an object and moves it, possibly altering its physical state. Agent-object and object-object interactions during a manipulation are a defining part of the performed action itself. In this thesis, we focus on extracting semantic cues, derived from geometric object interactions in 3D space during a manipulation, that are useful for action understanding at the cognitive level. First, we introduce a simple grounding model for the most common pairwise spatial relations between objects and investigate the descriptive power of their temporal evolution for action characterization. We propose a compact, abstract action descriptor that encodes the geometric object interactions during action execution, as captured by the spatial relation dynamics. Our experiments on a diverse dataset confirm both the validity and effectiveness of our spatial relation models and the discriminative power of our representation with respect to the underlying action semantics. Second, we model and detect lower level interactions, namely object contacts and separations, viewing them as topological scene changes within a dense motion estimation setting. In addition to improving motion estimation accuracy in the challenging case of motion boundaries induced by these events, our approach shows promising performance in the explicit detection and classification of the latter. Building upon dense motion estimation and using detected contact events as an attention mechanism, we propose a bottom-up pipeline for the guided segmentation and rigid motion extraction of manipulated objects. Finally, in addition to our methodological contributions, we introduce a new open-source software library for point cloud data processing, developed for the needs of this thesis, which aims at providing an easy to use, flexible, and efficient framework for the rapid development of performant software for a range of 3D perception tasks

A Survey on Deep Learning in Medical Image Registration: New Technologies, Uncertainty, Evaluation Metrics, and Beyond

Over the past decade, deep learning technologies have greatly advanced the field of medical image registration. The initial developments, such as ResNet-based and U-Net-based networks, laid the groundwork for deep learning-driven image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, and uncertainty estimation. These advancements have not only enriched the field of deformable image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration

Echocardiography

The book "Echocardiography - New Techniques" brings worldwide contributions from highly acclaimed clinical and imaging science investigators, and representatives from academic medical centers. Each chapter is designed and written to be accessible to those with a basic knowledge of echocardiography. Additionally, the chapters are meant to be stimulating and educational to the experts and investigators in the field of echocardiography. This book is aimed primarily at cardiology fellows on their basic echocardiography rotation, fellows in general internal medicine, radiology and emergency medicine, and experts in the arena of echocardiography. Over the last few decades, the rate of technological advancements has developed dramatically, resulting in new techniques and improved echocardiographic imaging. The authors of this book focused on presenting the most advanced techniques useful in today's research and in daily clinical practice. These advanced techniques are utilized in the detection of different cardiac pathologies in patients, in contributing to their clinical decision, as well as follow-up and outcome predictions. In addition to the advanced techniques covered, this book expounds upon several special pathologies with respect to the functions of echocardiography

Optical Flow At Occlusion Boundaries And In Occlusion Regions

Optical flow is an important research area in the Computer Vision field, with the estimation of optical flow at occlusion still an open problem. Traditional approaches to this problem have either used additional terms in a regularization calculation (the flow still tends to “bleed” across occlusion boundaries) or a local least squares calculation that attempted to minimize the influence of two adjacent differently moving regions on the optical flow at points close to both regions (the flow still tends to be “corrupted” by the two regions). Ideally, optical flow for two adjacent differently moving regions should be distinct right up to the occlusion boundary. A recent approach to calculate optical flow at occlusion is to combining boundary and region segmentation with the optical flow computation. Based on the work of Sundberg et al. Arbelaez et al. and Brox et al., we implement a motion gradient (mg) edge map algorithm which detects motion information in closed regions in the image sequences. Here we utilize the motion gradient as an additional local cue in the globalized probability of a boundary (gPb) as a new boundary detector to produce a gPb + mg contour map. The next step is to apply the Ultrametric Contour Map (UCM) mechanism, which is a framework to compute closed contours in a hierarchical region tree to produce a hierarchical edge map which indicates possible boundaries, including occlusion boundaries. We implemented Sundberg et al.’s work to detect occlusion boundaries using optical flow, but, unlike Sundberg et al., we compute and display optical flow everywhere. The Sundberg et al. optical flow was generated by Brox et al’s method. They used a least squares calcula- tion on the brox flow at pixels around an occlusion boundary to determine whether a boundary computed by the gPb − UCM library developed by UC Berkeley is occluding or occluded. We extended their least squares idea to 1st and 2nd order optical flow models to generate dense opti- cal flow inside each closed region. Finally, we analyze our optical flow fields both qualitatively and quantitatively. In particular, for quantitative analysis, we use warping error, as the correct flow is unknown. We show improved results over those of Sundberg et al., note a number of shortcomings in Sundberg et al.’s approach and point to areas of future research

Analysis of Image Sequence Data with Applications to Two-Dimensional Echocardiography

Digital two-dimensional echocardiography is an ultrasonic imaging technique that is used as an increasingly important noninvasive technique in the comprehensive characterization of the left ventricular structure and function. Quantitative analysis often uses heart wall motion and other shape attributes such as the heart wall thickness, heart chamber area, and the variation of these attributes throughout the cardiac cycle. These analyses require the complete determination of the heart wall boundaries. Poor image quality and large amount of noise makes computer detection of the boundaries difficult. An algorithm to detect both the inner and outer heart wall boundaries is presented. The algorithm was applied to images acquired from animal studies and from a tissue equivalent phantom to verify the performance. Different approaches to exploiting the temporal redundancy of the image data without making use of results from image segmentation and scene interpretation are explored. A new approach to perform image flow analysis is developed based on the Total Least Squares method. The result of this processing is an estimate of the velocities in the image plane. In an image understanding system, information acquired from related domains by other sensors are often useful to the analysis of images. Electrocardiogram signals measure the change of electrical potential changes in the heart muscle an d provide important information such as the timing data for image sequence analysis. These signals are frequently plagued by impulsive muscle noise and background drift due to patient movement. A new approach to solving these problems is presented using mathematical morphology. Experiments addressing various aspects of the problem, such as algorithm performance, choice of operator parameters, and response to sinusoidal inputs, are reported