Estimating and understanding motion : from diagnostic to robotic surgery

Estimating and understanding motion from an image sequence is a central topic in computer vision. The high interest in this topic is because we are living in a world where many events that occur in the environment are dynamic. This makes motion estimation and understanding a natural component and a key factor in a widespread of applications including object recognition , 3D shape reconstruction, autonomous navigation and medica! diagnosis. Particularly, we focus on the medical domain in which understanding the human body for clinical purposes requires retrieving the organs' complex motion patterns, which is in general a hard problem when using only image data. In this thesis, we cope with this problem by posing the question - How to achieve a realistic motion estimation to offer a better clinical understanding? We focus this thesis on answering this question by using a variational formulation as a basis to understand one of the most complex motions in the human's body, the heart motion, through three different applications: (i) cardiac motion estimation for diagnostic, (ii) force estimation and (iii) motion prediction, both for robotic surgery. Firstly, we focus on a central topic in cardiac imaging that is the estimation of the cardiac motion. The main aim is to offer objective and understandable measures to physicians for helping them in the diagnostic of cardiovascular diseases. We employ ultrafast ultrasound data and tools for imaging motion drawn from diverse areas such as low-rank analysis and variational deformation to perform a realistic cardiac motion estimation. The significance is that by taking low-rank data with carefully chosen penalization, synergies in this complex variational problem can be created. We demonstrate how our proposed solution deals with complex deformations through careful numerical experiments using realistic and simulated data. We then move from diagnostic to robotic surgeries where surgeons perform delicate procedures remotely through robotic manipulators without directly interacting with the patients. As a result, they lack force feedback, which is an important primary sense for increasing surgeon-patient transparency and avoiding injuries and high mental workload. To solve this problem, we follow the conservation principies of continuum mechanics in which it is clear that the change in shape of an elastic object is directly proportional to the force applied. Thus, we create a variational framework to acquire the deformation that the tissues undergo due to an applied force. Then, this information is used in a learning system to find the nonlinear relationship between the given data and the applied force. We carried out experiments with in-vivo and ex-vivo data and combined statistical, graphical and perceptual analyses to demonstrate the strength of our solution. Finally, we explore robotic cardiac surgery, which allows carrying out complex procedures including Off-Pump Coronary Artery Bypass Grafting (OPCABG). This procedure avoids the associated complications of using Cardiopulmonary Bypass (CPB) since the heart is not arrested while performing the surgery on a beating heart. Thus, surgeons have to deal with a dynamic target that compromisetheir dexterity and the surgery's precision. To compensate the heart motion, we propase a solution composed of three elements: an energy function to estimate the 3D heart motion, a specular highlight detection strategy and a prediction approach for increasing the robustness of the solution. We conduct evaluation of our solution using phantom and realistic datasets. We conclude the thesis by reporting our findings on these three applications and highlight the dependency between motion estimation and motion understanding at any dynamic event, particularly in clinical scenarios.L’estimació i comprensió del moviment dins d’una seqüència d’imatges és un tema central en la visió per ordinador, el que genera un gran interès perquè vivim en un entorn ple d’esdeveniments dinàmics. Per aquest motiu és considerat com un component natural i factor clau dins d’un ampli ventall d’aplicacions, el qual inclou el reconeixement d’objectes, la reconstrucció de formes tridimensionals, la navegació autònoma i el diagnòstic de malalties. En particular, ens situem en l’àmbit mèdic en el qual la comprensió del cos humà, amb finalitats clíniques, requereix l’obtenció de patrons complexos de moviment dels òrgans. Aquesta és, en general, una tasca difícil quan s’utilitzen només dades de tipus visual. En aquesta tesi afrontem el problema plantejant-nos la pregunta - Com es pot aconseguir una estimació realista del moviment amb l’objectiu d’oferir una millor comprensió clínica? La tesi se centra en la resposta mitjançant l’ús d’una formulació variacional com a base per entendre un dels moviments més complexos del cos humà, el del cor, a través de tres aplicacions: (i) estimació del moviment cardíac per al diagnòstic, (ii) estimació de forces i (iii) predicció del moviment, orientant-se les dues últimes en cirurgia robòtica. En primer lloc, ens centrem en un tema principal en la imatge cardíaca, que és l’estimació del moviment cardíac. L’objectiu principal és oferir als metges mesures objectives i comprensibles per ajudar-los en el diagnòstic de les malalties cardiovasculars. Fem servir dades d’ultrasons ultraràpids i eines per al moviment d’imatges procedents de diverses àrees, com ara l’anàlisi de baix rang i la deformació variacional, per fer una estimació realista del moviment cardíac. La importància rau en que, en prendre les dades de baix rang amb una penalització acurada, es poden crear sinergies en aquest problema variacional complex. Mitjançant acurats experiments numèrics, amb dades realístiques i simulades, hem demostrat com les nostres propostes solucionen deformacions complexes. Després passem del diagnòstic a la cirurgia robòtica, on els cirurgians realitzen procediments delicats remotament, a través de manipuladors robòtics, sense interactuar directament amb els pacients. Com a conseqüència, no tenen la percepció de la força com a resposta, que és un sentit primari important per augmentar la transparència entre el cirurgià i el pacient, per evitar lesions i per reduir la càrrega de treball mental. Resolem aquest problema seguint els principis de conservació de la mecànica del medi continu, en els quals està clar que el canvi en la forma d’un objecte elàstic és directament proporcional a la força aplicada. Per això hem creat un marc variacional que adquireix la deformació que pateixen els teixits per l’aplicació d’una força. Aquesta informació s’utilitza en un sistema d’aprenentatge, per trobar la relació no lineal entre les dades donades i la força aplicada. Hem dut a terme experiments amb dades in-vivo i ex-vivo i hem combinat l’anàlisi estadístic, gràfic i de percepció que demostren la robustesa de la nostra solució. Finalment, explorem la cirurgia cardíaca robòtica, la qual cosa permet realitzar procediments complexos, incloent la cirurgia coronària sense bomba (off-pump coronary artery bypass grafting o OPCAB). Aquest procediment evita les complicacions associades a l’ús de circulació extracorpòria (Cardiopulmonary Bypass o CPB), ja que el cor no s’atura mentre es realitza la cirurgia. Això comporta que els cirurgians han de tractar amb un objectiu dinàmic que compromet la seva destresa i la precisió de la cirurgia. Per compensar el moviment del cor, proposem una solució composta de tres elements: un funcional d’energia per estimar el moviment tridimensional del cor, una estratègia de detecció de les reflexions especulars i una aproximació basada en mètodes de predicció, per tal d’augmentar la robustesa de la solució. L’avaluació de la nostra solució s’ha dut a terme mitjançant conjunts de dades sintètiques i realistes. La tesi conclou informant dels nostres resultats en aquestes tres aplicacions i posant de relleu la dependència entre l’estimació i la comprensió del moviment en qualsevol esdeveniment dinàmic, especialment en escenaris clínics.Postprint (published version

DEVELOPMENT AND IMPLEMENTATION OF NOVEL STRATEGIES TO EXPLOIT 3D ULTRASOUND IMAGING IN CARDIOVASCULAR COMPUTATIONAL BIOMECHANICS

Introduction In the past two decades, major advances have been made in cardiovascular diseases assessment and treatment owing to the advent of sophisticated and more accurate imaging techniques, allowing for better understanding the complexity of 3D anatomical cardiovascular structures1. Volumetric acquisition enables the visualization of cardiac districts from virtually any perspective, better appreciating patient-specific anatomical complexity, as well as an accurate quantitative functional evaluation of chamber volumes and mass avoiding geometric assumptions2. Additionally, this scenario also allowed the evolution from generic to patient-specific 3D cardiac models that, based on in vivo imaging, faithfully represent the anatomy and different cardiac features of a given alive subject, being pivotal either in diagnosis and in planning guidance3. Precise morphological and functional knowledge about either the heart valves\u2019 apparatus and the surrounding structures is crucial when dealing with diagnosis as well as preprocedural planning4. To date, computed tomography (CT) and real-time 3D echocardiography (rt3DE) are typically exploited in this scenario since they allow for encoding comprehensive structural and dynamic information even in the fourth dimension (i.e., time)5,6. However, owing to its cost-effectiveness and very low invasiveness, 3D echocardiography has become the method of choice in most situations for performing the evaluation of cardiac function, developing geometrical models which can provide quantitative anatomical assessment7. Complementing this scenario, computational models have been introduced as numerical engineering tools aiming at adding qualitative and quantitative information on the biomechanical behavior in terms of stress-strain response and other multifactorial parameters8. In particular, over the two last decades, their applications have been ranging from elucidating the heart biomechanics underlying different patho-physiological conditions9 to predicting the effects of either surgical or percutaneous procedures, even comparing several implantation techniques and devices10. At the early stage, most of the studies focused on FE modeling in cardiac environment were based on paradigmatic models11\u201315, being mainly exploited to explore and investigate biomechanical alterations following a specific pathological scenario or again to better understand whether a surgical treatment is better or worse than another one. Differently, nowadays the current generation of computational models heavily exploits the detailed anatomical information yielded by medical imaging to provide patient-specific analyses, paving the way toward the development of virtual surgical-planning tools16\u201319. In this direction, cardiac magnetic resonance (CMR) and CT/micro-CT are the mostly accomplished imaging modality, since they can provide well-defined images thanks to their spatial and temporal resolutions20\u201325. Nonetheless, they cannot be applied routinely in clinical practice, as it can be differently done with rt3DE, progressively became the modality of choice26 since it has no harmful effects on the patient and no radiopaque contrast agent is needed. Despite these advantages, 3D volumetric ultrasound imaging shows intrinsic limitations beyond its limited resolution: i) the deficiency of morphological detail owing to either not so easy achievable detection (e.g., tricuspid valve) or not proper acoustic window, ii) the challenge of tailoring computational models to the patient-specific scenario mimicking the morphology as well as the functionality of the investigated cardiac district (e.g., tethering effect exerted by chordal apparatus in mitral valve insufficiency associated to left ventricular dilation), and iii) the needing to systematically analyse devices performances when dealing with real-life cases where ultrasound imaging is the only performable technique but lacking of standardized acquisition protocol. Main findings In the just described scenario, the main aim of this work was focused on the implementation, development and testing of numerical strategies in order to overcome issues when dealing with 3D ultrasound imaging exploitation towards predictive patient-specific modelling approaches focused on both morphological and biomechanical analyses. Specifically, the first specific objective was the development of a novel approach integrating in vitro imaging and finite element (FE) modeling to evaluate tricuspid valve (TV) biomechanics, facing with the lack of information on anatomical features owing to the clinically evident demanding detection of this anatomical district through in vivo imaging. \u2022 An innovative and semi-automated framework was implemented to generate 3D model of TV, to quantitively describe its 3D morphology and to assess its biomechanical behaviour. At this aim, an image-based in vitro experimental approach was integrated with numerical models based on FE strategy. Experimental measurements directly performed on the benchmark (mock circulation loop) were compared with geometrical features computed on the 3D reconstructed model, pinpointing a global good consistency. Furthermore, obtained realistic reconstructions were used as the input of the FE models, even accounting for proper description of TV leaflets\u2019 anisotropic mechanical response. As done experimentally, simulations reproduced both \u201cincompetent\u201d (FTR) and \u201ccompetent-induced\u201d (PMA), proving the efficiency of such a treatment and suggesting translational potential to the clinic. The second specific aim was the implementation of a computational framework able to reproduce a functionally equivalent model of the mitral valve (MV) sub-valvular apparatus through chordae tendineae topology optimization, aiming at chordae rest length arrangement to be able to include their pre-stress state associated to specific ventricular conformation. \u2022 We sought to establish a framework to build geometrically tractable, functionally equivalent models of the MV chordae tendineae, addressing one of the main topics of the computational scientific literature towards the development of faithful patient-specific models from in vivo imaging. Exploiting the mass spring model (MSM) approach, an iterative tool was proposed aiming to the topology optimization of a paradigmatic chordal apparatus of MVs affected by functional regurgitation, in order to be able to equivalently account for tethering effect exerted by the chordae themselves. The results have shown that the algorithm actually lowered the error between the simulated valve and ground truth data, although the intensity of this improvement is strongly valve-dependent.Finally, the last specific aim was the creation of a numerical strategy able to allow for patient-specific geometrical reconstruction both pre- and post- LVAD implantation, in a specific high-risk clinical scenario being rt3DE the only available imaging technique to be used but without any acquisition protocol. \u2022 We proposed a numerical approach which allowed for a systematic and selective analysis of the mechanism associated to intraventricular thrombus formation and thrombogenic complications in a LVAD-treated dilated left ventricle (LV). Ad-hoc geometry reconstruction workflow was implemented to overcome limitations associated to imaging acquisition in this specific scenario, thus being able to generate computational model of the LV assisted with LVAD. In details, results suggested that blood stasis is influenced either by LVAD flow rate and, to a greater extent, by LV residual contractility, being the positioning of the inflow cannula insertion mandatory to be considered when dealing with LVAD thrombogenic potential assessment

Analysis of cardiac motion using MRI and nonrigid image registration

Imperial Users onl

Modelling the Human Cardiac Fluid Mechanics. 4th ed

With the Karlsruhe Heart Model (KaHMo) we aim to share our vision of integrated computational simulation across multiple disciplines of cardiovascular research, and emphasis yet again the importance of Modelling the Human Cardiac Fluid Mechanics within the framework of the international STICH study. The focus of this work is on integrated cardiovascular fluid mechanics, and the potential benefits to future cardiovascular research and the wider bio-medical community

Feature based estimation of myocardial motion from tagged MR images

In the past few years we witnessed an increase in mortality due to cancer relative to mortality due to cardiovascular diseases. In 2008, the Netherlands Statistics Agency reports that 33.900 people died of cancer against 33.100 deaths due to cardiovascular diseases, making cancer the number one cause of death in the Netherlands [33]. Even if the rate of people affected by heart diseases is continually rising, they "simply don’t die of it", according to the research director Prof. Mat Daemen of research institute CARIM of the University of Maastricht [50]. The reason for this is the early diagnosis, and the treatment of people with identified risk factors for diseases like ischemic heart disease, hypertrophic cardiomyopathy, thoracic aortic disease, pericardial (sac around the heart) disease, cardiac tumors, pulmonary artery disease, valvular disease, and congenital heart disease before and after surgical repair. Cardiac imaging plays a crucial role in the early diagnosis, since it allows the accurate investigation of a large amount of imaging data in a small amount of time. Moreover, cardiac imaging reduces costs of inpatient care, as has been shown in recent studies [77]. With this in mind, in this work we have provided several tools with the aim to help the investigation of the cardiac motion. In chapters 2 and 3 we have explored a novel variational optic flow methodology based on multi-scale feature points to extract cardiac motion from tagged MR images. Compared to constant brightness methods, this new approach exhibits several advantages. Although the intensity of critical points is also influenced by fading, critical points do retain their characteristic even in the presence of intensity changes, such as in MR imaging. In an experiment in section 5.4 we have applied this optic flow approach directly on tagged MR images. A visual inspection confirmed that the extracted motion fields realistically depicted the cardiac wall motion. The method exploits also the advantages from the multiscale framework. Because sparse velocity formulas 2.9, 3.7, 6.21, and 7.5 provide a number of equations equal to the number of unknowns, the method does not suffer from the aperture problem in retrieving velocities associated to the critical points. In chapters 2 and 3 we have moreover introduced a smoothness component of the optic flow equation described by means of covariant derivatives. This is a novelty in the optic flow literature. Many variational optic flow methods present a smoothness component that penalizes for changes from global assumptions such as isotropic or anisotropic smoothness. In the smoothness term proposed deviations from a predefined motion model are penalized. Moreover, the proposed optic flow equation has been decomposed in rotation-free and divergence-free components. This decomposition allows independent tuning of the two components during the vector field reconstruction. The experiments and the Table of errors provided in 3.8 showed that the combination of the smoothness term, influenced by a predefined motion model, and the Helmholtz decomposition in the optic flow equation reduces the average angular error substantially (20%-25%) with respect to a similar technique that employs only standard derivatives in the smoothness term. In section 5.3 we extracted the motion field of a phantom of which we know the ground truth of and compared the performance of this optic flow method with the performance of other optic flow methods well known in the literature, such as the Horn and Schunck [76] approach, the Lucas and Kanade [111] technique and the tuple image multi-scale optic flow constraint equation of Van Assen et al. [163]. Tests showed that the proposed optic flow methodology provides the smallest average angular error (AAE = 3.84 degrees) and L2 norm = 0.1. In this work we employed the Helmholtz decomposition also to study the cardiac behavior, since the vector field decomposition allows to investigate cardiac contraction and cardiac rotation independently. In chapter 4 we carried out an analysis of cardiac motion of ten volunteers and one patient where we estimated the kinetic energy for the different components. This decomposition is useful since it allows to visualize and quantify the contributions of each single vector field component to the heart beat. Local measurements of the kinetic energy have also been used to detect areas of the cardiac walls with little movement. Experiments on a patient and a comparison between a late enhancement cardiac image and an illustration of the cardiac kinetic energy on a bull’s eye plot illustrated that a correspondence between an infarcted area and an area with very small kinetic energy exists. With the aim to extend in the future the proposed optic flow equation to a 3D approach, in chapter 6 we investigated the 3D winding number approach as a tool to locate critical points in volume images. We simplified the mathematics involved with respect to a previous work [150] and we provided several examples and applications such as cardiac motion estimation from 3-dimensional tagged images, follicle and neuronal cell counting. Finally in chapter 7 we continued our investigation on volume tagged MR images, by retrieving the cardiac motion field using a 3-dimensional and simple version of the proposed optic flow equation based on standard derivatives. We showed that the retrieved motion fields display the contracting and rotating behavior of the cardiac muscle. We moreover extracted the through-plane component, which provides a realistic illustration of the vector field and is missed by 2-dimensional approaches

Constrained manifold learning for the characterization of pathological deviations from normality

International audienceThis paper describes a technique to (1) learn the representation of a pathological motion pattern from a given population, and (2) compare individuals to this population. Our hypothesis is that this pattern can be modeled as a deviation from normal motion by means of non-linear embedding techniques. Each subject is represented by a 2D map of local motion abnormalities, obtained from a statistical atlas of myocardial motion built from a healthy population. The algorithm estimates a manifold from a set of patients with varying degrees of the same disease, and compares individuals to the training population using a mapping to the manifold and a distance to normality along the manifold. The approach extends recent manifold learning techniques by constraining the manifold to pass by a physiologically meaningful origin representing a normal motion pattern. Interpolation techniques using locally adjustable kernel improve the accuracy of the method. The technique is applied in the context of cardiac resynchronization therapy (CRT), focusing on a specific motion pattern of intra-ventricular dyssynchrony called septal flash (SF). We estimate the manifold from 50 CRT candidates with SF and test it on 37 CRT candidates and 21 healthy volunteers. Experiments highlight the relevance of nonlinear techniques to model a pathological pattern from the training set and compare new individuals to this pattern

Ultrafast Ultrasound Imaging

Among medical imaging modalities, such as computed tomography (CT) and magnetic resonance imaging (MRI), ultrasound imaging stands out due to its temporal resolution. Owing to the nature of medical ultrasound imaging, it has been used for not only observation of the morphology of living organs but also functional imaging, such as blood flow imaging and evaluation of the cardiac function. Ultrafast ultrasound imaging, which has recently become widely available, significantly increases the opportunities for medical functional imaging. Ultrafast ultrasound imaging typically enables imaging frame-rates of up to ten thousand frames per second (fps). Due to the extremely high temporal resolution, this enables visualization of rapid dynamic responses of biological tissues, which cannot be observed and analyzed by conventional ultrasound imaging. This Special Issue includes various studies of improvements to the performance of ultrafast ultrasoun