Fully Automatic and Real-Time Catheter Segmentation in X-Ray Fluoroscopy

Augmenting X-ray imaging with 3D roadmap to improve guidance is a common strategy. Such approaches benefit from automated analysis of the X-ray images, such as the automatic detection and tracking of instruments. In this paper, we propose a real-time method to segment the catheter and guidewire in 2D X-ray fluoroscopic sequences. The method is based on deep convolutional neural networks. The network takes as input the current image and the three previous ones, and segments the catheter and guidewire in the current image. Subsequently, a centerline model of the catheter is constructed from the segmented image. A small set of annotated data combined with data augmentation is used to train the network. We trained the method on images from 182 X-ray sequences from 23 different interventions. On a testing set with images of 55 X-ray sequences from 5 other interventions, a median centerline distance error of 0.2 mm and a median tip distance error of 0.9 mm was obtained. The segmentation of the instruments in 2D X-ray sequences is performed in a real-time fully-automatic manner.Comment: Accepted to MICCAI 201

End-to-End Real-time Catheter Segmentation with Optical Flow-Guided Warping during Endovascular Intervention

Accurate real-time catheter segmentation is an important pre-requisite for robot-assisted endovascular intervention. Most of the existing learning-based methods for catheter segmentation and tracking are only trained on small-scale datasets or synthetic data due to the difficulties of ground-truth annotation. Furthermore, the temporal continuity in intraoperative imaging sequences is not fully utilised. In this paper, we present FW-Net, an end-to-end and real-time deep learning framework for endovascular intervention. The proposed FW-Net has three modules: a segmentation network with encoder-decoder architecture, a flow network to extract optical flow information, and a novel flow-guided warping function to learn the frame-to-frame temporal continuity. We show that by effectively learning temporal continuity, the network can successfully segment and track the catheters in real-time sequences using only raw ground-truth for training. Detailed validation results confirm that our FW-Net outperforms state-of-the-art techniques while achieving real-time performance.Comment: ICRA 202

Dynamic Analysis of X-ray Angiography for Image-Guided Coronary Interventions

Percutaneous coronary intervention (PCI) is a minimally-invasive procedure for treating patients with coronary artery disease. PCI is typically performed with image guidance using X-ray angiograms (XA) in which coronary arter

Improved Image Guidance in TACE Procedures

Purpose of the work in this thesis is to improve the image guidance in TACE procedures. More specifically, we intend to develop and evaluate technology that permits dynamic roadmapping based on a 3D model of the liver vasculature

Image-Based Force Estimation and Haptic Rendering For Robot-Assisted Cardiovascular Intervention

Clinical studies have indicated that the loss of haptic perception is the prime limitation of robot-assisted cardiovascular intervention technology, hindering its global adoption. It causes compromised situational awareness for the surgeon during the intervention and may lead to health risks for the patients. This doctoral research was aimed at developing technology for addressing the limitation of the robot-assisted intervention technology in the provision of haptic feedback. The literature review showed that sensor-free force estimation (haptic cue) on endovascular devices, intuitive surgeon interface design, and haptic rendering within the surgeon interface were the major knowledge gaps. For sensor-free force estimation, first, an image-based force estimation methods based on inverse finite-element methods (iFEM) was developed and validated. Next, to address the limitation of the iFEM method in real-time performance, an inverse Cosserat rod model (iCORD) with a computationally efficient solution for endovascular devices was developed and validated. Afterward, the iCORD was adopted for analytical tip force estimation on steerable catheters. The experimental studies confirmed the accuracy and real-time performance of the iCORD for sensor-free force estimation. Afterward, a wearable drift-free rotation measurement device (MiCarp) was developed to facilitate the design of an intuitive surgeon interface by decoupling the rotation measurement from the insertion measurement. The validation studies showed that MiCarp had a superior performance for spatial rotation measurement compared to other modalities. In the end, a novel haptic feedback system based on smart magnetoelastic elastomers was developed, analytically modeled, and experimentally validated. The proposed haptics-enabled surgeon module had an unbounded workspace for interventional tasks and provided an intuitive interface. Experimental validation, at component and system levels, confirmed the usability of the proposed methods for robot-assisted intervention systems

ADVANCED INTRAVASCULAR MAGNETIC RESONANCE IMAGING WITH INTERACTION

Intravascular (IV) Magnetic Resonance Imaging (MRI) is a specialized class of interventional MRI (iMRI) techniques that acquire MRI images through blood vessels to guide, identify and/or treat pathologies inside the human body which are otherwise difficult to locate and treat precisely. Here, interactions based on real-time computations and feedback are explored to improve the accuracy and efficiency of IVMRI procedures. First, an IV MRI-guided high-intensity focused ultrasound (HIFU) ablation method is developed for targeting perivascular pathology with minimal injury to the vessel wall. To take advantage of real-time feedback, a software interface is developed for monitoring thermal dose with real-time MRI thermometry, and an MRI-guided ablation protocol developed and tested on muscle and liver tissue ex vivo. It is shown that, with cumulative thermal dose monitored with MRI thermometry, lesion location and dimensions can be estimated consistently, and desirable thermal lesions can be achieved in animals in vivo. Second, to achieve fully interactive IV MRI, high-resolution real-time 10 frames-per-second (fps) MRI endoscopy is developed as an advance over prior methods of MRI endoscopy. Intravascular transmit-receive MRI endoscopes are fabricated for highly under-sampled radial-projection MRI in a clinical 3Tesla MRI scanner. Iterative nonlinear reconstruction is accelerated using graphics processor units (GPU) to achieve true real-time endoscopy visualization at the scanner. The results of high-speed MRI endoscopy at 6-10 fps are consistent with fully-sampled MRI endoscopy and histology, with feasibility demonstrated in vivo in a large animal model. Last, a general framework for automatic imaging contrast tuning over MRI protocol parameters is explored. The framework reveals typical signal patterns over different protocol parameters from calibration imaging data and applies this knowledge to design efficient acquisition strategies and predicts contrasts under unacquired protocols. An external computer in real-time communication with the MRI console is utilized for online processing and controlling MRI acquisitions. This workflow enables machine learning for optimizing acquisition strategies in general, and provides a foundation for efficiently tuning MRI protocol parameters to perform interventional MRI in the highly varying and interactive environments commonly in play. This work is loosely inspired by prior research on extremely accelerated MRI relaxometry using the minimal-acquisition linear algebraic modeling (SLAM) method

Exploiting Temporal Image Information in Minimally Invasive Surgery

Minimally invasive procedures rely on medical imaging instead of the surgeons direct vision. While preoperative images can be used for surgical planning and navigation, once the surgeon arrives at the target site real-time intraoperative imaging is needed. However, acquiring and interpreting these images can be challenging and much of the rich temporal information present in these images is not visible. The goal of this thesis is to improve image guidance for minimally invasive surgery in two main areas. First, by showing how high-quality ultrasound video can be obtained by integrating an ultrasound transducer directly into delivery devices for beating heart valve surgery. Secondly, by extracting hidden temporal information through video processing methods to help the surgeon localize important anatomical structures. Prototypes of delivery tools, with integrated ultrasound imaging, were developed for both transcatheter aortic valve implantation and mitral valve repair. These tools provided an on-site view that shows the tool-tissue interactions during valve repair. Additionally, augmented reality environments were used to add more anatomical context that aids in navigation and in interpreting the on-site video. Other procedures can be improved by extracting hidden temporal information from the intraoperative video. In ultrasound guided epidural injections, dural pulsation provides a cue in finding a clear trajectory to the epidural space. By processing the video using extended Kalman filtering, subtle pulsations were automatically detected and visualized in real-time. A statistical framework for analyzing periodicity was developed based on dynamic linear modelling. In addition to detecting dural pulsation in lumbar spine ultrasound, this approach was used to image tissue perfusion in natural video and generate ventilation maps from free-breathing magnetic resonance imaging. A second statistical method, based on spectral analysis of pixel intensity values, allowed blood flow to be detected directly from high-frequency B-mode ultrasound video. Finally, pulsatile cues in endoscopic video were enhanced through Eulerian video magnification to help localize critical vasculature. This approach shows particular promise in identifying the basilar artery in endoscopic third ventriculostomy and the prostatic artery in nerve-sparing prostatectomy. A real-time implementation was developed which processed full-resolution stereoscopic video on the da Vinci Surgical System

Flow Control and MRI-Compatible Particle Injector: Application to Magnetic Resonance Navigation

Notre groupe de recherche travaille sur une technique de navigation par résonance magnétique (NRM) qui vise à améliorer l’efficacité du ciblage des médicaments vers les zones tumorales. Cette technique a été subdivisée en cinq parties : 1. Conception des microparticules. La taille et les matériaux constituants les particules doivent répondre aux exigences médicales et physiologiques de l'embolisation humaine, ainsi qu'à la faisabilité du pilotage des microparticules en utilisant les gradients magnétiques (≤ 40 mT/m) d'un système clinique d'imagerie par résonance magnétique (IRM). 2. Contrôle du flot sanguin. Un contrôle du flot sanguin a été mis au point pour permettre une navigation suffisamment rapide afin de réduire le temps d'injection tout en étant suffisamment lente pour assurer un taux de réussite élevé pour la NRM. 3. Conception d’un injecteur pour la formation d'agrégats de microparticules de tailles contrôlables. Un injecteur IRM compatible a été conçu pour permettre l’injection d’agrégats de particules pour à la fois réduire le temps d'injection et augmenter l'efficacité de la NRM en raison du volume magnétique plus important injecté à chaque fois. 4. Logiciel NRM. Une séquence NRM est utilisée pour suivre et orienter les agrégats injectés. 5. Intégration de l'injecteur de particules, du contrôle du flot sanguin et du logiciel NRM. Dans cette thèse, nous cherchons à trouver des solutions aux étapes 2, 3, 4 et 5. L’auteur a constaté que la combinaison d’un micro-flot vibratoire et d’un flot constant à faible vitesse pouvait rendre la vitesse de dérive des particules basse et constante. Un système de contrôle de l'écoulement composé d'une machine vibratoire générant le flot en question et d'une pompe péristaltique a été conçu et fabriqué pour générer ces deux types d'écoulement. Ensuite, le système a été intégré au NRM pour tester les manipulations in vitro visées. Dans une IRM de 1.5 Tesla (T), les microparticules encapsulant des nanoparticules superparamagnétiques ont été navigués dans un canal à bifurcation unique en forme de Y. En comparant les résultats avec le NRM à débit constant, nous avons démontré que le modèle de flot vibrantoire proposé peut améliorer de manière significative le taux de réussite du NRM avec un gradient magnétique inférieur à 40 mT/m qui correspond au seuil maximum de gradient qui peut être utilisé sur une IRM clinique traditionnelle. Par la suite, un injecteur de particules compatible avec l'IRM, composé de deux pompes péristaltiques, d'un compteur optique et d'un piège magnétique, a été proposé pour former des agrégats de particules de taille spécifique. Afin de déterminer la conception et la configuration optimales de l'injecteur, les propriétés magnétiques des microparticules, la compatibilité magnétique des différentes pièces de l'injecteur et la distribution spatiale du champ magnétique du système IRM ont été étudiées de manière exhaustive. Les particules utilisées dans l'essai avaient un diamètre de 230 ± 35 μm, ce qui respecte les spécifications requises pour une chimioembolisation trans-artérielle (TACE) chez l'adulte. Nous avons démontré que l’injecteur pouvait former des agrégats contenant 20 à 60 microparticules avec une précision de 6 particules. Les agrégats ayant des longueurs globales correspondantes de 1.6 à 3.2 mm, ce qui se situe juste dans l’échelle des diamètres internes des artères hépatiques propres et des branches de division droite et gauche. Par la suite, des agrégats constitués de 25 particules ont été injectés dans un fantôme imitant des conditions physiologiques et rhéologiques humaine. Dans ce cas, 82% des agrégats (n = 50) ont réussi à atteindre les sous-branches ciblées. Enfin, nous avons démontré qu'il était possible d'intégrer le flot vibrant combiné avec flot constant, l'injecteur et la séquence NRM à notre injecteur afin d'établir une synchronisation entre la formation, la propulsion, la navigation et le suivi de bolus de particules dans un fantôme avec deux niveaux de bifurcation. Un modèle théorique de la taille et l'orientation des vaisseaux a été étudié et pris en compte lors du calcul de la longueur appropriée de l'agrégat de particules pour différentes tailles de vaisseaux. Une séquence d’IRM rapide (True FISP) et un gradient magnétique de 20 mT/m ont été choisis pour suivre et orienter les agrégats. Les particules magnétiques de 200 μm de diamètre moyen ont été utilisées pour évaluer l'efficacité de la NRM avec la méthode proposée. Dans les expériences, sur la base du modèle théorique, la longueur totale des agrégats a été fixée à environ 1.6 mm. Lorsqu'un agrégat était prêt, il était injecté dans le fantôme situé au centre du tunnel de l’IRM, imitant des situations réelles. Pendant ce temps, un signal de déclenchement généré automatiquement par le générateur déclenche la séquence NRM. Les agrégats de particules ont été entraînés par le flot combiné et dirigés puis suivis par la séquence NRM. En fonction de la position des agrégats dans le fantôme, la direction du gradient de navigation a été ajustée pour diriger les agrégats de microparticules dans la branche ciblée. Lorsque le tube principal du fantôme était parallèle à B0, la distribution de base des agrégats sans NRM de gauche à gauche (GG), de gauche à droite (GD), de droite à gauche (DG) et de droite à droite (DD) viii était de 4%, 96%, 0% et 0% respectivement. La précision a atteint 84% (GG), 100% (GD), 84% (DG) et 96% (DD) (P < 0.001, P = 1.0, P < 0.001, 1, P <0.001) en utilisant la séquence de NRM correspondantes pour diriger chaque agrégat dans une branche ciblée. Ensuite, le fantôme a subi une rotation de 90 degrés horizontalement. Dans cette configuration, la branche D-G qui avait le plus faible ratio de distribution de base de 0%, passait à 80% (P <0.001) après NRM. De plus, le taux de réussite du MRN était toujours supérieur à 92% à la première bifurcation dans les expériences mentionnées ci-dessus. En conclusion, ce projet a proposé un nouveau modèle d'écoulement pour augmenter le taux de réussite de la NRM avec un gradient magnétique de 40 mT/m. Il s'agit d'une étape importante pour les expériences in vivo utilisant le système d'IRM clinique. Ensuite, un injecteur compatible avec l'IRM, capable de contrôler la taille des agrégats de particules, a été conçu et testé. Enfin, la première intégration du système d’injection de particules, qui alterne un gradient de guidage et une séquence True FISP dans un logiciel NRM dédié, confirme que le NRM peut être utilisée pour naviguer in vitro des agrégats de particules à travers deux niveaux de bifurcations à l’aide d’une IRM clinique 3 T sans modification matérielle.----------ABSTRACT The author’s research group has been working on a technique of magnetic resonance navigation (MRN) which aims to improve the targeting efficiency of drugs towards tumour areas. This technique was subdivided mainly into the following steps: 1. Particle design. Sizes and materials of particles need to meet the medical and physiological requirements of human embolization, as well as the feasibility of steering the particles by using magnetic gradients (≤ 40 mT/m) of a clinical magnetic resonance imaging (MRI) system. 2. Flow control. The particles drifted by blood flow must be fast enough to decrease the particle injection time while being appropriately slow to ensure a high success rate for MRN. 3. The conception of a dedicated MR compatible injector to create microparticle aggregates with controllable sizes. The formation of aggregates can both decrease the injection time and increase the MRN efficiency because of the larger magnetic volume injected each time. 4. MRN software. An MRN sequence is used to track and steer the injected aggregates. 5. Integration of the particle injector, flow control and the MRN software. In this thesis, I aim to find solutions for steps 2, 3, 4 and 5. The author found that the combination of micro-vibrating flow and low-velocity constant flow could make the velocity of the drifted particles low and steady. A flow control system consisting of a vibrator and a peristaltic pump was designed and fabricated to generate these two flow patterns. Then, the system was integrated with MRN to test for the targeted in vitro manipulations. In a 1.5 Tesla (T) MRI system, microparticles encapsulating superparamagnetic nanoparticles were navigated in a Y-shaped single bifurcation channel. By comparing the results with MRN with constant flow, I demonstrated that the proposed flow pattern can significantly improve the success rate of MRN under a magnetic gradient of 40 mT/m, a force that can be obtained but is difficult to increase further when using a traditional clinical MRI system. Subsequently, an MRI-compatible particle injector, composed of two peristaltic pumps, an optical counter and a magnetic trap was proposed to form specific-sized particle aggregates. In order to determine the optimal design and setup of the injector, the magnetic property of microparticles, the magnetic compatibility of different parts within the injector and the field distribution of the MRI system were studied comprehensively. The particles used in the test had diameters of 230 ± 35 μm which respect the specifications needed for trans-arterial chemoembolization (TACE) in human adults. I demonstrated that the system could form aggregates containing 20 to 60 microparticles with a precision of 6 particles. The corresponding aggregate lengths ranged from 1.6 to 3.2 mm, which is just within the scale of internal diameters of the common, right and left hepatic arteries. Subsequently, aggregates consisting of 25 particles were injected into a phantom which mimics realistic physiological and rheological conditions. Under such circumstances, 82% of the aggregates (n = 50) were able to successfully reach subbranches. At last, I demonstrated the feasibility of integrating the combined flow pattern, the injector and the MRN sequence to establish synchronization between the formation, propulsion, steering and tracking of particle boluses in a two-level bifurcation phantom. To start with the establishment of a theoretical model, the size and orientation of the vessels were comprehensively studied and took into consideration when the calculation for the appropriate length of the particle aggregate for different vessel sizes. Next, a steady-state coherent sequence (True FISP) and a 20 mT/m magnetic gradient were chosen as the MRN sequence and force used to track and steer moving aggregates. Finally, magnetic particles of 200 μm mean diameter were used to evaluate the MRN efficiency of the proposed method. In the experiments, based on the theoretical model, the aggregate length was set, through the injector, to roughly 1.2 mm. When an aggregate was ready, it was injected into the phantom located in the MRI bore, imitating real-life situations. Meanwhile, a trigger signal automatically generated by the trigger generator would start the MRN sequence. Particle aggregates were drifted by the combined flow and were steered and tracked by the MRN sequence. According to the position of the aggregates in the phantom, the direction of the steering gradient would be tuned to ensure that the particles were steered into the targeted branch. When the main tube of the phantom was parallel to B0, the left–left (L-L), left–right (L-R), right–left (R-L) and right–right (R-R) baseline distribution of aggregates with no MRN were 4%, 96%, 0% and 0% respectively. The accuracy reached 84% (L-L), 100%(L-R), 84% (R-L) and 96% (R-R) (P < 0.001, P = 1.0, P < 0.001, P < 0.001) after applying corresponding MRN operations to steer each aggregate into a targeted branch. Then, the phantom was rotated 90 degrees horizontally. In that setup, the RL branch had the smallest baseline distribution ratio of 0%, which increased to 80% (P < 0.001) through MRN. Moreover, the success rate of MRN was always more than 92% at the 1st bifurcation in the experiments above. In conclusion, this project proposes a new flow pattern for increasing the MRN success rate under the magnetic gradient of 40 mT/m. This is an important step for in vivo experiments using the clinical MRI system. Then, an MRI-compatible injector, capable of controlling the size of particle aggregates, was designed and tested. At last, the first integration of the particle injection system which interleaves a steering gradient and a True FISP sequence in a dedicated MRN software confirmed that MRN can be used to navigate particle aggregates in vitro across two branch divisions in a 3 T clinical MRI system without hardware modification

A Clinician's Contribution to Biomedical Engineering in Experimental Echocardiography

The research of this thesis has been focused on the biomedical engineering aspects of new techniques of echocardiography. In close collaboration with the engineers of the Experimental Echocardiography Department of the Thoraxcentre, Erasmus University, Rotterdam, new methods to measure coronary blood flow and arterial wall elasticity with intravascular ultrasound (IVUS) have been developed. We have also investigated the clinical application of these measurements and have tried to improve traditional techniques based on intracoronary Doppler wires. In another field, we have developed a method to determine the radiation dose delivered in the wall of coronary arteries treated with brachytherapy. in collaboration with the Emory University, Atlanta, GA. This method utilizes 3-dimensional IVUS reconstruction combined with radiotherapy treatment planning. Finally, the tools developed for the recording of the signals of intracoronary Doppler wires have been adapted, during a stay at the Cleveland Clinic Foundation, OK for the study of left ventricular mechanics and the compliance of the large arteries. This has been achieved by simultaneous acquisition of non-invasive pressure (with tonometry) and flow (with transthoracic Doppler echocardiography) signals. The fruits of an old and close collaboration with the Institute Biomedical Technology of the Ghent University can also be found in different chapters. This work is subdivided in five major parts, and a detailed introductory chapter precedes each one