Retinal Fundus Image Registration via Vascular Structure Graph Matching

Motivated by the observation that a retinal fundus image may contain some unique geometric structures within its vascular trees which can be utilized for feature matching, in this paper, we proposed a graph-based registration framework called GM-ICP to align pairwise retinal images. First, the retinal vessels are automatically detected and represented as vascular structure graphs. A graph matching is then performed to find global correspondences between vascular bifurcations. Finally, a revised ICP algorithm incorporating with quadratic transformation model is used at fine level to register vessel shape models. In order to eliminate the incorrect matches from global correspondence set obtained via graph matching, we proposed a structure-based sample consensus (STRUCT-SAC) algorithm. The advantages of our approach are threefold: (1) global optimum solution can be achieved with graph matching; (2) our method is invariant to linear geometric transformations; and (3) heavy local feature descriptors are not required. The effectiveness of our method is demonstrated by the experiments with 48 pairs retinal images collected from clinical patients

Coronary Artery Segmentation and Motion Modelling

Conventional coronary artery bypass surgery requires invasive sternotomy and the use of a cardiopulmonary bypass, which leads to long recovery period and has high infectious potential. Totally endoscopic coronary artery bypass (TECAB) surgery based on image guided robotic surgical approaches have been developed to allow the clinicians to conduct the bypass surgery off-pump with only three pin holes incisions in the chest cavity, through which two robotic arms and one stereo endoscopic camera are inserted. However, the restricted field of view of the stereo endoscopic images leads to possible vessel misidentification and coronary artery mis-localization. This results in 20-30% conversion rates from TECAB surgery to the conventional approach. We have constructed patient-specific 3D + time coronary artery and left ventricle motion models from preoperative 4D Computed Tomography Angiography (CTA) scans. Through temporally and spatially aligning this model with the intraoperative endoscopic views of the patient's beating heart, this work assists the surgeon to identify and locate the correct coronaries during the TECAB precedures. Thus this work has the prospect of reducing the conversion rate from TECAB to conventional coronary bypass procedures. This thesis mainly focus on designing segmentation and motion tracking methods of the coronary arteries in order to build pre-operative patient-specific motion models. Various vessel centreline extraction and lumen segmentation algorithms are presented, including intensity based approaches, geometric model matching method and morphology-based method. A probabilistic atlas of the coronary arteries is formed from a group of subjects to facilitate the vascular segmentation and registration procedures. Non-rigid registration framework based on a free-form deformation model and multi-level multi-channel large deformation diffeomorphic metric mapping are proposed to track the coronary motion. The methods are applied to 4D CTA images acquired from various groups of patients and quantitatively evaluated

Automatic Spatiotemporal Analysis of Cardiac Image Series

RÉSUMÉ À ce jour, les maladies cardiovasculaires demeurent au premier rang des principales causes de décès en Amérique du Nord. Chez l’adulte et au sein de populations de plus en plus jeunes, la soi-disant épidémie d’obésité entraînée par certaines habitudes de vie tels que la mauvaise alimentation, le manque d’exercice et le tabagisme est lourde de conséquences pour les personnes affectées, mais aussi sur le système de santé. La principale cause de morbidité et de mortalité chez ces patients est l’athérosclérose, une accumulation de plaque à l’intérieur des vaisseaux sanguins à hautes pressions telles que les artères coronaires. Les lésions athérosclérotiques peuvent entraîner l’ischémie en bloquant la circulation sanguine et/ou en provoquant une thrombose. Cela mène souvent à de graves conséquences telles qu’un infarctus. Outre les problèmes liés à la sténose, les parois artérielles des régions criblées de plaque augmentent la rigidité des parois vasculaires, ce qui peut aggraver la condition du patient. Dans la population pédiatrique, la pathologie cardiovasculaire acquise la plus fréquente est la maladie de Kawasaki. Il s’agit d’une vasculite aigüe pouvant affecter l’intégrité structurale des parois des artères coronaires et mener à la formation d’anévrismes. Dans certains cas, ceux-ci entravent l’hémodynamie artérielle en engendrant une perfusion myocardique insuffisante et en activant la formation de thromboses. Le diagnostic de ces deux maladies coronariennes sont traditionnellement effectués à l’aide d’angiographies par fluoroscopie. Pendant ces examens paracliniques, plusieurs centaines de projections radiographiques sont acquises en séries suite à l’infusion artérielle d’un agent de contraste. Ces images révèlent la lumière des vaisseaux sanguins et la présence de lésions potentiellement pathologiques, s’il y a lieu. Parce que les séries acquises contiennent de l’information très dynamique en termes de mouvement du patient volontaire et involontaire (ex. battements cardiaques, respiration et déplacement d’organes), le clinicien base généralement son interprétation sur une seule image angiographique où des mesures géométriques sont effectuées manuellement ou semi-automatiquement par un technicien en radiologie. Bien que l’angiographie par fluoroscopie soit fréquemment utilisé partout dans le monde et souvent considéré comme l’outil de diagnostic “gold-standard” pour de nombreuses maladies vasculaires, la nature bidimensionnelle de cette modalité d’imagerie est malheureusement très limitante en termes de spécification géométrique des différentes régions pathologiques. En effet, la structure tridimensionnelle des sténoses et des anévrismes ne peut pas être pleinement appréciée en 2D car les caractéristiques observées varient selon la configuration angulaire de l’imageur. De plus, la présence de lésions affectant les artères coronaires peut ne pas refléter la véritable santé du myocarde, car des mécanismes compensatoires naturels (ex. vaisseaux----------ABSTRACT Cardiovascular disease continues to be the leading cause of death in North America. In adult and, alarmingly, ever younger populations, the so-called obesity epidemic largely driven by lifestyle factors that include poor diet, lack of exercise and smoking, incurs enormous stresses on the healthcare system. The primary cause of serious morbidity and mortality for these patients is atherosclerosis, the build up of plaque inside high pressure vessels like the coronary arteries. These lesions can lead to ischemic disease and may progress to precarious blood flow blockage or thrombosis, often with infarction or other severe consequences. Besides the stenosis-related outcomes, the arterial walls of plaque-ridden regions manifest increased stiffness, which may exacerbate negative patient prognosis. In pediatric populations, the most prevalent acquired cardiovascular pathology is Kawasaki disease. This acute vasculitis may affect the structural integrity of coronary artery walls and progress to aneurysmal lesions. These can hinder the blood flow’s hemodynamics, leading to inadequate downstream perfusion, and may activate thrombus formation which may lead to precarious prognosis. Diagnosing these two prominent coronary artery diseases is traditionally performed using fluoroscopic angiography. Several hundred serial x-ray projections are acquired during selective arterial infusion of a radiodense contrast agent, which reveals the vessels’ luminal area and possible pathological lesions. The acquired series contain highly dynamic information on voluntary and involuntary patient movement: respiration, organ displacement and heartbeat, for example. Current clinical analysis is largely limited to a single angiographic image where geometrical measures will be performed manually or semi-automatically by a radiological technician. Although widely used around the world and generally considered the gold-standard diagnosis tool for many vascular diseases, the two-dimensional nature of this imaging modality is limiting in terms of specifying the geometry of various pathological regions. Indeed, the 3D structures of stenotic or aneurysmal lesions may not be fully appreciated in 2D because their observable features are dependent on the angular configuration of the imaging gantry. Furthermore, the presence of lesions in the coronary arteries may not reflect the true health of the myocardium, as natural compensatory mechanisms may obviate the need for further intervention. In light of this, cardiac magnetic resonance perfusion imaging is increasingly gaining attention and clinical implementation, as it offers a direct assessment of myocardial tissue viability following infarction or suspected coronary artery disease. This type of modality is plagued, however, by motion similar to that present in fluoroscopic imaging. This issue predisposes clinicians to laborious manual intervention in order to align anatomical structures in sequential perfusion frames, thus hindering automation o

Automatic Spatiotemporal Analysis of Cardiac Image Series

RÉSUMÉ À ce jour, les maladies cardiovasculaires demeurent au premier rang des principales causes de décès en Amérique du Nord. Chez l’adulte et au sein de populations de plus en plus jeunes, la soi-disant épidémie d’obésité entraînée par certaines habitudes de vie tels que la mauvaise alimentation, le manque d’exercice et le tabagisme est lourde de conséquences pour les personnes affectées, mais aussi sur le système de santé. La principale cause de morbidité et de mortalité chez ces patients est l’athérosclérose, une accumulation de plaque à l’intérieur des vaisseaux sanguins à hautes pressions telles que les artères coronaires. Les lésions athérosclérotiques peuvent entraîner l’ischémie en bloquant la circulation sanguine et/ou en provoquant une thrombose. Cela mène souvent à de graves conséquences telles qu’un infarctus. Outre les problèmes liés à la sténose, les parois artérielles des régions criblées de plaque augmentent la rigidité des parois vasculaires, ce qui peut aggraver la condition du patient. Dans la population pédiatrique, la pathologie cardiovasculaire acquise la plus fréquente est la maladie de Kawasaki. Il s’agit d’une vasculite aigüe pouvant affecter l’intégrité structurale des parois des artères coronaires et mener à la formation d’anévrismes. Dans certains cas, ceux-ci entravent l’hémodynamie artérielle en engendrant une perfusion myocardique insuffisante et en activant la formation de thromboses. Le diagnostic de ces deux maladies coronariennes sont traditionnellement effectués à l’aide d’angiographies par fluoroscopie. Pendant ces examens paracliniques, plusieurs centaines de projections radiographiques sont acquises en séries suite à l’infusion artérielle d’un agent de contraste. Ces images révèlent la lumière des vaisseaux sanguins et la présence de lésions potentiellement pathologiques, s’il y a lieu. Parce que les séries acquises contiennent de l’information très dynamique en termes de mouvement du patient volontaire et involontaire (ex. battements cardiaques, respiration et déplacement d’organes), le clinicien base généralement son interprétation sur une seule image angiographique où des mesures géométriques sont effectuées manuellement ou semi-automatiquement par un technicien en radiologie. Bien que l’angiographie par fluoroscopie soit fréquemment utilisé partout dans le monde et souvent considéré comme l’outil de diagnostic “gold-standard” pour de nombreuses maladies vasculaires, la nature bidimensionnelle de cette modalité d’imagerie est malheureusement très limitante en termes de spécification géométrique des différentes régions pathologiques. En effet, la structure tridimensionnelle des sténoses et des anévrismes ne peut pas être pleinement appréciée en 2D car les caractéristiques observées varient selon la configuration angulaire de l’imageur. De plus, la présence de lésions affectant les artères coronaires peut ne pas refléter la véritable santé du myocarde, car des mécanismes compensatoires naturels (ex. vaisseaux----------ABSTRACT Cardiovascular disease continues to be the leading cause of death in North America. In adult and, alarmingly, ever younger populations, the so-called obesity epidemic largely driven by lifestyle factors that include poor diet, lack of exercise and smoking, incurs enormous stresses on the healthcare system. The primary cause of serious morbidity and mortality for these patients is atherosclerosis, the build up of plaque inside high pressure vessels like the coronary arteries. These lesions can lead to ischemic disease and may progress to precarious blood flow blockage or thrombosis, often with infarction or other severe consequences. Besides the stenosis-related outcomes, the arterial walls of plaque-ridden regions manifest increased stiffness, which may exacerbate negative patient prognosis. In pediatric populations, the most prevalent acquired cardiovascular pathology is Kawasaki disease. This acute vasculitis may affect the structural integrity of coronary artery walls and progress to aneurysmal lesions. These can hinder the blood flow’s hemodynamics, leading to inadequate downstream perfusion, and may activate thrombus formation which may lead to precarious prognosis. Diagnosing these two prominent coronary artery diseases is traditionally performed using fluoroscopic angiography. Several hundred serial x-ray projections are acquired during selective arterial infusion of a radiodense contrast agent, which reveals the vessels’ luminal area and possible pathological lesions. The acquired series contain highly dynamic information on voluntary and involuntary patient movement: respiration, organ displacement and heartbeat, for example. Current clinical analysis is largely limited to a single angiographic image where geometrical measures will be performed manually or semi-automatically by a radiological technician. Although widely used around the world and generally considered the gold-standard diagnosis tool for many vascular diseases, the two-dimensional nature of this imaging modality is limiting in terms of specifying the geometry of various pathological regions. Indeed, the 3D structures of stenotic or aneurysmal lesions may not be fully appreciated in 2D because their observable features are dependent on the angular configuration of the imaging gantry. Furthermore, the presence of lesions in the coronary arteries may not reflect the true health of the myocardium, as natural compensatory mechanisms may obviate the need for further intervention. In light of this, cardiac magnetic resonance perfusion imaging is increasingly gaining attention and clinical implementation, as it offers a direct assessment of myocardial tissue viability following infarction or suspected coronary artery disease. This type of modality is plagued, however, by motion similar to that present in fluoroscopic imaging. This issue predisposes clinicians to laborious manual intervention in order to align anatomical structures in sequential perfusion frames, thus hindering automation o

Deep Learning in Cardiology

The medical field is creating large amount of data that physicians are unable to decipher and use efficiently. Moreover, rule-based expert systems are inefficient in solving complicated medical tasks or for creating insights using big data. Deep learning has emerged as a more accurate and effective technology in a wide range of medical problems such as diagnosis, prediction and intervention. Deep learning is a representation learning method that consists of layers that transform the data non-linearly, thus, revealing hierarchical relationships and structures. In this review we survey deep learning application papers that use structured data, signal and imaging modalities from cardiology. We discuss the advantages and limitations of applying deep learning in cardiology that also apply in medicine in general, while proposing certain directions as the most viable for clinical use.Comment: 27 pages, 2 figures, 10 table

Relation between plaque type, plaque thickness, blood shear stress, and plaque stress in coronary arteries assessed by X-ray Angiography and Intravascular Ultrasound

Purpose: Atheromatic plaque progression is affected, among others phenomena, by biomechanical, biochemical, and physiological factors. In this paper, the authors introduce a novel framework able to provide both morphological (vessel radius, plaque thickness, and type) and biomechanical (wall shear stress and Von Mises stress) indices of coronary arteries. Methods: First, the approach reconstructs the three-dimensional morphology of the vessel from intravascular ultrasound(IVUS) and Angiographic sequences, requiring minimal user interaction. Then, a computational pipeline allows to automatically assess fluid-dynamic and mechanical indices. Ten coronary arteries are analyzed illustrating the capabilities of the tool and confirming previous technical and clinical observations. Results: The relations between the arterial indices obtained by IVUS measurement and simulations have been quantitatively analyzed along the whole surface of the artery, extending the analysis of the coronary arteries shown in previous state of the art studies. Additionally, for the first time in the literature, the framework allows the computation of the membrane stresses using a simplified mechanical model of the arterial wall. Conclusions: Circumferentially (within a given frame), statistical analysis shows an inverse relation between the wall shear stress and the plaque thickness. At the global level (comparing a frame within the entire vessel), it is observed that heavy plaque accumulations are in general calcified and are located in the areas of the vessel having high wall shear stress. Finally, in their experiments the inverse proportionality between fluid and structural stresses is observed

Inferring Geodesic Cerebrovascular Graphs: Image Processing, Topological Alignment and Biomarkers Extraction

A vectorial representation of the vascular network that embodies quantitative features - location, direction, scale, and bifurcations - has many potential neuro-vascular applications. Patient-specific models support computer-assisted surgical procedures in neurovascular interventions, while analyses on multiple subjects are essential for group-level studies on which clinical prediction and therapeutic inference ultimately depend. This first motivated the development of a variety of methods to segment the cerebrovascular system. Nonetheless, a number of limitations, ranging from data-driven inhomogeneities, the anatomical intra- and inter-subject variability, the lack of exhaustive ground-truth, the need for operator-dependent processing pipelines, and the highly non-linear vascular domain, still make the automatic inference of the cerebrovascular topology an open problem. In this thesis, brain vessels’ topology is inferred by focusing on their connectedness. With a novel framework, the brain vasculature is recovered from 3D angiographies by solving a connectivity-optimised anisotropic level-set over a voxel-wise tensor field representing the orientation of the underlying vasculature. Assuming vessels joining by minimal paths, a connectivity paradigm is formulated to automatically determine the vascular topology as an over-connected geodesic graph. Ultimately, deep-brain vascular structures are extracted with geodesic minimum spanning trees. The inferred topologies are then aligned with similar ones for labelling and propagating information over a non-linear vectorial domain, where the branching pattern of a set of vessels transcends a subject-specific quantized grid. Using a multi-source embedding of a vascular graph, the pairwise registration of topologies is performed with the state-of-the-art graph matching techniques employed in computer vision. Functional biomarkers are determined over the neurovascular graphs with two complementary approaches. Efficient approximations of blood flow and pressure drop account for autoregulation and compensation mechanisms in the whole network in presence of perturbations, using lumped-parameters analog-equivalents from clinical angiographies. Also, a localised NURBS-based parametrisation of bifurcations is introduced to model fluid-solid interactions by means of hemodynamic simulations using an isogeometric analysis framework, where both geometry and solution profile at the interface share the same homogeneous domain. Experimental results on synthetic and clinical angiographies validated the proposed formulations. Perspectives and future works are discussed for the group-wise alignment of cerebrovascular topologies over a population, towards defining cerebrovascular atlases, and for further topological optimisation strategies and risk prediction models for therapeutic inference. Most of the algorithms presented in this work are available as part of the open-source package VTrails

Deep Representation Learning with Limited Data for Biomedical Image Synthesis, Segmentation, and Detection

Biomedical imaging requires accurate expert annotation and interpretation that can aid medical staff and clinicians in automating differential diagnosis and solving underlying health conditions. With the advent of Deep learning, it has become a standard for reaching expert-level performance in non-invasive biomedical imaging tasks by training with large image datasets. However, with the need for large publicly available datasets, training a deep learning model to learn intrinsic representations becomes harder. Representation learning with limited data has introduced new learning techniques, such as Generative Adversarial Networks, Semi-supervised Learning, and Self-supervised Learning, that can be applied to various biomedical applications. For example, ophthalmologists use color funduscopy (CF) and fluorescein angiography (FA) to diagnose retinal degenerative diseases. However, fluorescein angiography requires injecting a dye, which can create adverse reactions in the patients. So, to alleviate this, a non-invasive technique needs to be developed that can translate fluorescein angiography from fundus images. Similarly, color funduscopy and optical coherence tomography (OCT) are also utilized to semantically segment the vasculature and fluid build-up in spatial and volumetric retinal imaging, which can help with the future prognosis of diseases. Although many automated techniques have been proposed for medical image segmentation, the main drawback is the model's precision in pixel-wise predictions. Another critical challenge in the biomedical imaging field is accurately segmenting and quantifying dynamic behaviors of calcium signals in cells. Calcium imaging is a widely utilized approach to studying subcellular calcium activity and cell function; however, large datasets have yielded a profound need for fast, accurate, and standardized analyses of calcium signals. For example, image sequences from calcium signals in colonic pacemaker cells ICC (Interstitial cells of Cajal) suffer from motion artifacts and high periodic and sensor noise, making it difficult to accurately segment and quantify calcium signal events. Moreover, it is time-consuming and tedious to annotate such a large volume of calcium image stacks or videos and extract their associated spatiotemporal maps. To address these problems, we propose various deep representation learning architectures that utilize limited labels and annotations to address the critical challenges in these biomedical applications. To this end, we detail our proposed semi-supervised, generative adversarial networks and transformer-based architectures for individual learning tasks such as retinal image-to-image translation, vessel and fluid segmentation from fundus and OCT images, breast micro-mass segmentation, and sub-cellular calcium events tracking from videos and spatiotemporal map quantification. We also illustrate two multi-modal multi-task learning frameworks with applications that can be extended to other domains of biomedical applications. The main idea is to incorporate each of these as individual modules to our proposed multi-modal frameworks to solve the existing challenges with 1) Fluorescein angiography synthesis, 2) Retinal vessel and fluid segmentation, 3) Breast micro-mass segmentation, and 4) Dynamic quantification of calcium imaging datasets