Automated analysis of 3D echocardiography

In this thesis we aim at automating the analysis of 3D echocardiography, mainly targeting the functional analysis of the left ventricle. Manual analysis of these data is cumbersome, time-consuming and is associated with inter-observer and inter-institutional variability. Methods for reconstruction of 3D echocardiographic images from fast rotating ultrasound transducers is presented and methods for analysis of 3D echocardiography in general, using tracking, detection and model-based segmentation techniques to ultimately fully automatically segment the left ventricle for functional analysis. We show that reliable quantification of left ventricular volume and mitral valve displacement can be achieved using the presented techniques.SenterNovem (IOP Beeldverwerking, grant IBVC02003), Dutch Technology Foundation STW (grant 06666)UBL - phd migration 201

Automated Analysis of 3D Stress Echocardiography

__Abstract__ The human circulatory system consists of the heart, blood, arteries, veins and capillaries. The heart is the muscular organ which pumps the blood through the human body (Fig. 1.1,1.2). Deoxygenated blood flows through the right atrium into the right ventricle, which pumps the blood into the pulmonary arteries. The blood is carried to the lungs, where it passes through a capillary network that enables the release of carbon dioxide and the uptake of oxygen. Oxygenated blood then returns to the heart via the pulmonary veins and flows from the left atrium into the left ventricle. The left ventricle then pumps the blood through the aorta, the major artery which supplies blood to the rest of the body [Drake et a!., 2005; Guyton and Halt 1996]. Therefore, it is vital that the cardiovascular system remains healthy. Disease of the cardiovascular system, if untreated, ultimately leads to the failure of other organs and death

Preserving known anatomical topology in medical image segmentation using deep learning

Since the rise of deep learning, new medical image segmentation methods have rapidly been proposed with promising results, with each one reporting marginal improvements on the previous state-of-the-art (SOTA) method. However, on visual inspection, errors are often revealed, such as topological mistakes (e.g. holes or folds), that are not detected using traditional evaluation metrics, such as Dice. Moreover, correct topology is often essential in ensuring segmentations are anatomically and pathologically plausible and, ultimately, suitable for downstream image processing tasks. Therefore, there is a need for methods to focus on ensuring that the predicted segmentations are topologically correct, rather than just optimising the pixel-wise accuracy. In this thesis, I propose a method that utilises prior knowledge of anatomy to segment structures, whilst preserving the known topology. The presented model, Topology Encouraging Deformation Segmentation Network (TEDS-Net), performs segmentation by deforming a prior shape with the same topological features as the anatomy of interest using learnt topology-preserving deformation fields. However, here I show that such fields only guarantee topology preservation in the continuous domain and that their properties begin to break down when applied in discrete spaces. To overcome this effect, I introduced additional modifications in TEDS-Net to more strictly enforce topology preservation, a step often overlooked in previous work. Across this thesis, TEDS-Net is applied to a range of natural and medical image segmentation tasks. I show how it can be used for multiple topology types, multiple structures and in both two- and three-dimensions. Further, I show how TEDS-Net can be used to segment whole volumes using minimally annotated training data. Across these experiments, TEDS-Net outperforms all SOTA baselines on topology, whilst maintaining competitive pixel-wise accuracy. Finally, TEDS-Net is integrated into a whole medical imaging pipeline, to illustrate the importance of topologically correct segmentations for downstream tasks. TEDS-Net is used to segment the developing cortical plate from in-utero fetal brain ultrasound scans in 3D, to enable the characterisation of its complex growth and development during gestation. To the best of my knowledge, this task has only been previously attempted from magnetic resonance imaging (MRI), despite ultrasound being the modality of choice in prenatal care. This is likely due to large acoustic shadows obstructing key brain regions in ultrasound. Due to TEDS-Net anatomical constraints, it can anatomically guide the cortical plate segmentation in regions of shadows, producing a complete segmentation that enables accurate downstream analysis

Automated volume measurements in echocardiography by utilizing expert knowledge

Left ventricular (LV) volumes and ejection fraction (EF) are important parameters for diagnosis, prognosis, and treatment planning in patients with heart disease. These parameters are commonly measured by manual tracing in echocardiographic images, a procedure that is time consuming, prone to inter- and intra-observer variability, and require highly trained operators. This is particularly the case in three-dimensional (3D) echocardiography, where the increased amount of data makes manual tracing impractical. Automated methods for measuring LV volumes and EF can therefore improve efficiency and accuracy of echocardiographic examinations, giving better diagnosis at a lower cost. The main goal of this thesis was to improve the efficiency and quality of cardiac measurements. More specifically, the goal was to develop rapid and accurate methods that utilize expert knowledge for automated evaluation of cardiac function in echocardiography. The thesis presents several methods for automated volume and EF measurements in echocardiographic data. For two-dimensional (2D) echocardiography, an atlas based segmentation algorithm is presented in paper A. This method utilizes manually traced endocardial contours in a validated case database to control a snake optimized by dynamic programming. The challenge with this approach is to find the most optimal case in the database. More promising results are achieved in triplane echocardiography using a multiview and multi-frame extension to the active appearance model (AAM) framework, as demonstrated in paper B. The AAM generalizes better to new patient data and is based on more robust optimization schemes than the atlas-based method. In triplane images, the results of the AAM algorithm may be improved further by integrating a snake algorithm into the AAM framework and by constraining the AAM to manually defined landmarks, and this is shown in paper C. For 3D echocardiograms, a clinical semi-automated volume measurement tool with expert selected points is validated in paper D. This tool compares favorably to a reference measurement tool, with good agreement in measured volumes, and with a significantly lower analysis time. Finally, in paper E, fully automated real-time segmentation in 3D echocardiography is demonstrated using a 3D active shape model (ASM) of the left ventricle in a Kalman filter framework. The main advantage of this approach is its processing performance, allowing for real-time volume and EF estimates. Statistical models such as AAMs and ASMs provide elegant frameworks for incorporating expert knowledge into segmentation algorithms. Expert knowledge can also be utilized directly through manual input to semi-automated methods, allowing for manual initialization and correction of automatically determined volumes. The latter technique is particularly suitable for clinical routine examinations, while the fully automated 3D ASM method can extend the use of echocardiography to new clinical areas such as automated patient monitoring. In this thesis, different methods for utilizing expert knowledge in automated segmentation algorithms for echocardiography have been developed and evaluated. Particularly in 3D echocardiography, these contributions are expected to improve efficiency and quality of cardiac measurements

Bayesian Deep Learning for Cardiac Motion Modelling and Analysis

Cardiovascular diseases (CVDs) remain a primary cause of mortality globally, with an estimated 17.9 million deaths in 2019, accounting for 32% of all global fatalities. In recent decades, non-invasive imaging, particularly Magnetic Resonance Imaging (MRI), has become pivotal in diagnosing CVDs, offering high-resolution, multidimensional, and sequential cardiac data. However, the interpretation of cardiac MRI data is challenging, due to the complexities of cardiac motion and anatomical variations. Traditional manual methods are time-consuming and subject to variability. Deep learning (DL) methods, notably generative models, have recently advanced medical image analysis, offering state-of-the-art solutions for segmentation, registration, and motion modelling. This thesis encapsulates the development and validation of deep-learning frameworks in the field of cardiac motion modelling and analysis from sequential cardiac MRI scans. At its core, it introduces a probabilistic generative model for cardiac motion modelling, underpinned by temporal coherence, capable of synthesising new CMR sequences. Three models are derived from this foundational probabilistic model, each contributing to different aspects. Firstly, through the innovative application of gradient surgery techniques, we address the dual objectives of attaining high registration accuracy and ensuring the diffeomorphic characteristics of the predicted motion fields. Subsequently, we introduce the joint operation of ventricular segmentation and motion modelling. The proposed method combines anatomical precision with the dynamic temporal flow to enhance both the accuracy of motion modelling and the stability of sequential segmentation. Furthermore, we introduce a conditional motion transfer framework that leverages variational models for the generation of cardiac motion, enabling anomaly detection and the augmentation of data, particularly for pathologies that are less commonly represented in datasets. This capability to transfer and transform cardiac motion across healthy and pathological domains is set to revolutionize how clinicians and researchers understand and interpret cardiac function and anomalies. Collectively, these advancements present novelty and application potentials in cardiac image processing. The methodologies proposed herein have the potential to transform routine clinical diagnostics and interventions, allowing for more nuanced and detailed cardiac assessments. The probabilistic nature of these models promises to deliver not only more detailed insights into cardiac health but also to foster the development of personalised medicine approaches in cardiology

Estimating and understanding motion : from diagnostic to robotic surgery

Estimating and understanding motion from an image sequence is a central topic in computer vision. The high interest in this topic is because we are living in a world where many events that occur in the environment are dynamic. This makes motion estimation and understanding a natural component and a key factor in a widespread of applications including object recognition , 3D shape reconstruction, autonomous navigation and medica! diagnosis. Particularly, we focus on the medical domain in which understanding the human body for clinical purposes requires retrieving the organs' complex motion patterns, which is in general a hard problem when using only image data. In this thesis, we cope with this problem by posing the question - How to achieve a realistic motion estimation to offer a better clinical understanding? We focus this thesis on answering this question by using a variational formulation as a basis to understand one of the most complex motions in the human's body, the heart motion, through three different applications: (i) cardiac motion estimation for diagnostic, (ii) force estimation and (iii) motion prediction, both for robotic surgery. Firstly, we focus on a central topic in cardiac imaging that is the estimation of the cardiac motion. The main aim is to offer objective and understandable measures to physicians for helping them in the diagnostic of cardiovascular diseases. We employ ultrafast ultrasound data and tools for imaging motion drawn from diverse areas such as low-rank analysis and variational deformation to perform a realistic cardiac motion estimation. The significance is that by taking low-rank data with carefully chosen penalization, synergies in this complex variational problem can be created. We demonstrate how our proposed solution deals with complex deformations through careful numerical experiments using realistic and simulated data. We then move from diagnostic to robotic surgeries where surgeons perform delicate procedures remotely through robotic manipulators without directly interacting with the patients. As a result, they lack force feedback, which is an important primary sense for increasing surgeon-patient transparency and avoiding injuries and high mental workload. To solve this problem, we follow the conservation principies of continuum mechanics in which it is clear that the change in shape of an elastic object is directly proportional to the force applied. Thus, we create a variational framework to acquire the deformation that the tissues undergo due to an applied force. Then, this information is used in a learning system to find the nonlinear relationship between the given data and the applied force. We carried out experiments with in-vivo and ex-vivo data and combined statistical, graphical and perceptual analyses to demonstrate the strength of our solution. Finally, we explore robotic cardiac surgery, which allows carrying out complex procedures including Off-Pump Coronary Artery Bypass Grafting (OPCABG). This procedure avoids the associated complications of using Cardiopulmonary Bypass (CPB) since the heart is not arrested while performing the surgery on a beating heart. Thus, surgeons have to deal with a dynamic target that compromisetheir dexterity and the surgery's precision. To compensate the heart motion, we propase a solution composed of three elements: an energy function to estimate the 3D heart motion, a specular highlight detection strategy and a prediction approach for increasing the robustness of the solution. We conduct evaluation of our solution using phantom and realistic datasets. We conclude the thesis by reporting our findings on these three applications and highlight the dependency between motion estimation and motion understanding at any dynamic event, particularly in clinical scenarios.L’estimació i comprensió del moviment dins d’una seqüència d’imatges és un tema central en la visió per ordinador, el que genera un gran interès perquè vivim en un entorn ple d’esdeveniments dinàmics. Per aquest motiu és considerat com un component natural i factor clau dins d’un ampli ventall d’aplicacions, el qual inclou el reconeixement d’objectes, la reconstrucció de formes tridimensionals, la navegació autònoma i el diagnòstic de malalties. En particular, ens situem en l’àmbit mèdic en el qual la comprensió del cos humà, amb finalitats clíniques, requereix l’obtenció de patrons complexos de moviment dels òrgans. Aquesta és, en general, una tasca difícil quan s’utilitzen només dades de tipus visual. En aquesta tesi afrontem el problema plantejant-nos la pregunta - Com es pot aconseguir una estimació realista del moviment amb l’objectiu d’oferir una millor comprensió clínica? La tesi se centra en la resposta mitjançant l’ús d’una formulació variacional com a base per entendre un dels moviments més complexos del cos humà, el del cor, a través de tres aplicacions: (i) estimació del moviment cardíac per al diagnòstic, (ii) estimació de forces i (iii) predicció del moviment, orientant-se les dues últimes en cirurgia robòtica. En primer lloc, ens centrem en un tema principal en la imatge cardíaca, que és l’estimació del moviment cardíac. L’objectiu principal és oferir als metges mesures objectives i comprensibles per ajudar-los en el diagnòstic de les malalties cardiovasculars. Fem servir dades d’ultrasons ultraràpids i eines per al moviment d’imatges procedents de diverses àrees, com ara l’anàlisi de baix rang i la deformació variacional, per fer una estimació realista del moviment cardíac. La importància rau en que, en prendre les dades de baix rang amb una penalització acurada, es poden crear sinergies en aquest problema variacional complex. Mitjançant acurats experiments numèrics, amb dades realístiques i simulades, hem demostrat com les nostres propostes solucionen deformacions complexes. Després passem del diagnòstic a la cirurgia robòtica, on els cirurgians realitzen procediments delicats remotament, a través de manipuladors robòtics, sense interactuar directament amb els pacients. Com a conseqüència, no tenen la percepció de la força com a resposta, que és un sentit primari important per augmentar la transparència entre el cirurgià i el pacient, per evitar lesions i per reduir la càrrega de treball mental. Resolem aquest problema seguint els principis de conservació de la mecànica del medi continu, en els quals està clar que el canvi en la forma d’un objecte elàstic és directament proporcional a la força aplicada. Per això hem creat un marc variacional que adquireix la deformació que pateixen els teixits per l’aplicació d’una força. Aquesta informació s’utilitza en un sistema d’aprenentatge, per trobar la relació no lineal entre les dades donades i la força aplicada. Hem dut a terme experiments amb dades in-vivo i ex-vivo i hem combinat l’anàlisi estadístic, gràfic i de percepció que demostren la robustesa de la nostra solució. Finalment, explorem la cirurgia cardíaca robòtica, la qual cosa permet realitzar procediments complexos, incloent la cirurgia coronària sense bomba (off-pump coronary artery bypass grafting o OPCAB). Aquest procediment evita les complicacions associades a l’ús de circulació extracorpòria (Cardiopulmonary Bypass o CPB), ja que el cor no s’atura mentre es realitza la cirurgia. Això comporta que els cirurgians han de tractar amb un objectiu dinàmic que compromet la seva destresa i la precisió de la cirurgia. Per compensar el moviment del cor, proposem una solució composta de tres elements: un funcional d’energia per estimar el moviment tridimensional del cor, una estratègia de detecció de les reflexions especulars i una aproximació basada en mètodes de predicció, per tal d’augmentar la robustesa de la solució. L’avaluació de la nostra solució s’ha dut a terme mitjançant conjunts de dades sintètiques i realistes. La tesi conclou informant dels nostres resultats en aquestes tres aplicacions i posant de relleu la dependència entre l’estimació i la comprensió del moviment en qualsevol esdeveniment dinàmic, especialment en escenaris clínics.Postprint (published version

Deep Learning in Cardiac Magnetic Resonance Image Analysis and Cardiovascular Disease Diagnosis

Cardiovascular diseases (CVDs) are the leading cause of death in the world, accounting for 17.9 million deaths each year, 31\% of all global deaths. According to the World Health Organisation (WHO), this number is expected to rise to 23 million by 2030. As a noninvasive technique, medical imaging with corresponding computer vision techniques is becoming more and more popular for detecting, understanding, and analysing CVDs. With the advent of deep learning, there are significant improvements in medical image analysis tasks (e.g. image registration, image segmentation, mesh reconstruction from image), achieving much faster and more accurate registration, segmentation, reconstruction, and disease diagnosis. This thesis focuses on cardiac magnetic resonance images, systematically studying critical tasks in CVD analysis, including image registration, image segmentation, cardiac mesh reconstruction, and CVD prediction/diagnosis. We first present a thorough review of deep learning-based image registration approaches, and subsequently, propose a novel solution to the problem of discontinuity-preserving intra-subject cardiac image registration, which is generally ignored in previous deep learning-based registration methods. On the basis of this, a joint segmentation and registration framework is further proposed to learn the joint relationship between these two tasks, leading to better registration and segmentation performance. In order to characterise the shape and motion of the heart in 3D, we present a deep learning-based 3D mesh reconstruction network that is able to recover accurate 3D cardiac shapes from 2D slice-wise segmentation masks/contours in a fast and robust manner. Finally, for CVD prediction/diagnosis, we design a multichannel variational autoencoder to learn the joint latent representation of the original cardiac image and mesh, resulting in a shape-aware image representation (SAIR) that serves as an explainable biomarker. SAIR has been shown to outperform traditional biomarkers in the prediction of acute myocardial infarction and the diagnosis of several other CVDs, and can supplement existing biomarkers to improve overall predictive performance