INTERFACE DESIGN FOR A VIRTUAL REALITY-ENHANCED IMAGE-GUIDED SURGERY PLATFORM USING SURGEON-CONTROLLED VIEWING TECHNIQUES

Initiative has been taken to develop a VR-guided cardiac interface that will display and deliver information without affecting the surgeons’ natural workflow while yielding better accuracy and task completion time than the existing setup. This paper discusses the design process, the development of comparable user interface prototypes as well as an evaluation methodology that can measure user performance and workload for each of the suggested display concepts. User-based studies and expert recommendations are used in conjunction to es­ tablish design guidelines for our VR-guided surgical platform. As a result, a better understanding of autonomous view control, depth display, and use of virtual context, is attained. In addition, three proposed interfaces have been developed to allow a surgeon to control the view of the virtual environment intra-operatively. Comparative evaluation of the three implemented interface prototypes in a simulated surgical task scenario, revealed performance advantages for stereoscopic and monoscopic biplanar display conditions, as well as the differences between three types of control modalities. One particular interface prototype demonstrated significant improvement in task performance. Design recommendations are made for this interface as well as the others as we prepare for prospective development iterations

Computational Anatomy for Multi-Organ Analysis in Medical Imaging: A Review

The medical image analysis field has traditionally been focused on the development of organ-, and disease-specific methods. Recently, the interest in the development of more 20 comprehensive computational anatomical models has grown, leading to the creation of multi-organ models. Multi-organ approaches, unlike traditional organ-specific strategies, incorporate inter-organ relations into the model, thus leading to a more accurate representation of the complex human anatomy. Inter-organ relations are not only spatial, but also functional and physiological. Over the years, the strategies 25 proposed to efficiently model multi-organ structures have evolved from the simple global modeling, to more sophisticated approaches such as sequential, hierarchical, or machine learning-based models. In this paper, we present a review of the state of the art on multi-organ analysis and associated computation anatomy methodology. The manuscript follows a methodology-based classification of the different techniques 30 available for the analysis of multi-organs and multi-anatomical structures, from techniques using point distribution models to the most recent deep learning-based approaches. With more than 300 papers included in this review, we reflect on the trends and challenges of the field of computational anatomy, the particularities of each anatomical region, and the potential of multi-organ analysis to increase the impact of 35 medical imaging applications on the future of healthcare.Comment: Paper under revie

An image segmentation and registration approach to cardiac function analysis using MRI

Cardiovascular diseases (CVDs) are one of the major causes of death in the world. In recent years, significant progress has been made in the care and treatment of patients with such diseases. A crucial factor for this progress has been the development of magnetic resonance (MR) imaging which makes it possible to diagnose and assess the cardiovascular function of the patient. The ability to obtain high-resolution, cine volume images easily and safely has made it the preferred method for diagnosis of CVDs. MRI is also unique in its ability to introduce noninvasive markers directly into the tissue being imaged(MR tagging) during the image acquisition process. With the development of advanced MR imaging acquisition technologies, 3D MR imaging is more and more clinically feasible. This recent development has allowed new potentially 3D image analysis technologies to be deployed. However, quantitative analysis of cardiovascular system from the images remains a challenging topic. The work presented in this thesis describes the development of segmentation and motion analysis techniques for the study of the cardiac anatomy and function in cardiac magnetic resonance (CMR) images. The first main contribution of the thesis is the development of a fully automatic cardiac segmentation technique that integrates and combines a series of state-of-the-art techniques. The proposed segmentation technique is capable of generating an accurate 3D segmentation from multiple image sequences. The proposed segmentation technique is robust even in the presence of pathological changes, large anatomical shape variations and locally varying contrast in the images. Another main contribution of this thesis is the development of motion tracking techniques that can integrate motion information from different sources. For example, the radial motion of the myocardium can be tracked easily in untagged MR imaging since the epi- and endocardial surfaces are clearly visible. On the other hand, tagged MR imaging allows easy tracking of both longitudinal and circumferential motion. We propose a novel technique based on non-rigid image registration for the myocardial motion estimation using both untagged and 3D tagged MR images. The novel aspect of our technique is its simultaneous use of complementary information from both untagged and 3D tagged MR imaging. The similarity measure is spatially weighted to maximise the utility of information from both images. The thesis also proposes a sparse representation for free-form deformations (FFDs) using the principles of compressed sensing. The sparse free-form deformation (SFFD) model can capture fine local details such as motion discontinuities without sacrificing robustness. We demonstrate the capabilities of the proposed framework to accurately estimate smooth as well as discontinuous deformations in 2D and 3D CMR image sequences. Compared to the standard FFD approach, a significant increase in registration accuracy can be observed in datasets with discontinuous motion patterns. Both the segmentation and motion tracking techniques presented in this thesis have been applied to clinical studies. We focus on two important clinical applications that can be addressed by the techniques proposed in this thesis. The first clinical application aims at measuring longitudinal changes in cardiac morphology and function during the cardiac remodelling process. The second clinical application aims at selecting patients that positively respond to cardiac resynchronization therapy (CRT). The final chapter of this thesis summarises the main conclusions that can be drawn from the work presented here and also discusses possible avenues for future research

Label Efficient Deep Learning in Medical Imaging

Recent state-of-the-art deep learning frameworks require large, fully annotated training datasets that are, depending on the objective, time-consuming to generate. While in most fields, these labelling tasks can be parallelized massively or even outsourced, this is not the case for medical images. Usually, only a highly trained expert is able to generate these datasets. However, since additional manual annotation, especially for the purpose of segmentation or tracking, is typically not part of a radiologist's workflow, large and fully annotated datasets are a rare and scarce good. In this context, a variety of frameworks are proposed in this work to solve the problems that arise due to the lack of annotated training data across different medical imaging tasks and modalities. The first contribution as part of this thesis was to investigate weakly supervised learning on PET/CT data for the task of lesion segmentation. Using only class labels (tumor vs. no tumor), a classifier was first trained and subsequently used to generate Class Activation Maps highlighting regions with lesions. Based on these region proposals, final tumor segmentation could be performed with high accuracy in clinically relevant metrics. This drastically simplifies the process of training data generation, as only class labels have to be assigned to each slice of a scan instead of a full pixel-wise segmentation. To further reduce the time required to prepare training data, two self-supervised methods were investigated for the task of anatomical tissue segmentation and landmark detection. To this end, as a second contribution, a state-of-the-art tracking framework based on contrastive random walks was transferred, adapted and extended to the medical imaging domain. As contrastive learning often lacks real-time capability, a self-supervised template matching network was developed to address the task of real-time anatomical tissue tracking, yielding the third contribution of this work. Both of these methods have in common that only during inference the object or region of interest is defined, reducing the number of required labels to as few as one and allowing adaptation to different tasks without having to re-train or access the original training data. Despite the limited amount of labelled data, good results could be achieved for both tracking of organs across subjects as well as tissue tracking within time-series. State-of-the-art self-supervised learning in medical imaging is usually performed on 2D slices due to the lack of training data and limited computational resources. To exploit the three-dimensional structure of this type of data, self-supervised contrastive learning was performed on entire volumes using over 40,000 whole-body MRI scans forming the fourth contribution. Due to this pre-training, a large number of downstream tasks could be successfully addressed using only limited labelled data. Furthermore, the learned representations allows to visualize the entire dataset in a two-dimensional view. To encourage research in the field of automated lesion segmentation in PET/CT image data, the autoPET challenge was organized, which represents the fifth contribution

Modified mass-spring system for physically based deformation modeling

Mass-spring systems are considered the simplest and most intuitive of all deformable models. They are computationally efficient, and can handle large deformations with ease. But they suffer several intrinsic limitations. In this book a modified mass-spring system for physically based deformation modeling that addresses the limitations and solves them elegantly is presented. Several implementations in modeling breast mechanics, heart mechanics and for elastic images registration are presented

Segmentation of pelvic structures from preoperative images for surgical planning and guidance

Prostate cancer is one of the most frequently diagnosed malignancies globally and the second leading cause of cancer-related mortality in males in the developed world. In recent decades, many techniques have been proposed for prostate cancer diagnosis and treatment. With the development of imaging technologies such as CT and MRI, image-guided procedures have become increasingly important as a means to improve clinical outcomes. Analysis of the preoperative images and construction of 3D models prior to treatment would help doctors to better localize and visualize the structures of interest, plan the procedure, diagnose disease and guide the surgery or therapy. This requires efficient and robust medical image analysis and segmentation technologies to be developed. The thesis mainly focuses on the development of segmentation techniques in pelvic MRI for image-guided robotic-assisted laparoscopic radical prostatectomy and external-beam radiation therapy. A fully automated multi-atlas framework is proposed for bony pelvis segmentation in MRI, using the guidance of MRI AE-SDM. With the guidance of the AE-SDM, a multi-atlas segmentation algorithm is used to delineate the bony pelvis in a new \ac{MRI} where there is no CT available. The proposed technique outperforms state-of-the-art algorithms for MRI bony pelvis segmentation. With the SDM of pelvis and its segmented surface, an accurate 3D pelvimetry system is designed and implemented to measure a comprehensive set of pelvic geometric parameters for the examination of the relationship between these parameters and the difficulty of robotic-assisted laparoscopic radical prostatectomy. This system can be used in both manual and automated manner with a user-friendly interface. A fully automated and robust multi-atlas based segmentation has also been developed to delineate the prostate in diagnostic MR scans, which have large variation in both intensity and shape of prostate. Two image analysis techniques are proposed, including patch-based label fusion with local appearance-specific atlases and multi-atlas propagation via a manifold graph on a database of both labeled and unlabeled images when limited labeled atlases are available. The proposed techniques can achieve more robust and accurate segmentation results than other multi-atlas based methods. The seminal vesicles are also an interesting structure for therapy planning, particularly for external-beam radiation therapy. As existing methods fail for the very onerous task of segmenting the seminal vesicles, a multi-atlas learning framework via random decision forests with graph cuts refinement has further been proposed to solve this difficult problem. Motivated by the performance of this technique, I further extend the multi-atlas learning to segment the prostate fully automatically using multispectral (T1 and T2-weighted) MR images via hybrid \ac{RF} classifiers and a multi-image graph cuts technique. The proposed method compares favorably to the previously proposed multi-atlas based prostate segmentation. The work in this thesis covers different techniques for pelvic image segmentation in MRI. These techniques have been continually developed and refined, and their application to different specific problems shows ever more promising results.Open Acces

Modified mass-spring system for physically based deformation modeling

Mass-spring systems are considered the simplest and most intuitive of all deformable models. They are computationally efficient, and can handle large deformations with ease. But they suffer several intrinsic limitations. In this book a modified mass-spring system for physically based deformation modeling that addresses the limitations and solves them elegantly is presented. Several implementations in modeling breast mechanics, heart mechanics and for elastic images registration are presented

Automatic Spatiotemporal Analysis of Cardiac Image Series

RÉSUMÉ À ce jour, les maladies cardiovasculaires demeurent au premier rang des principales causes de décès en Amérique du Nord. Chez l’adulte et au sein de populations de plus en plus jeunes, la soi-disant épidémie d’obésité entraînée par certaines habitudes de vie tels que la mauvaise alimentation, le manque d’exercice et le tabagisme est lourde de conséquences pour les personnes affectées, mais aussi sur le système de santé. La principale cause de morbidité et de mortalité chez ces patients est l’athérosclérose, une accumulation de plaque à l’intérieur des vaisseaux sanguins à hautes pressions telles que les artères coronaires. Les lésions athérosclérotiques peuvent entraîner l’ischémie en bloquant la circulation sanguine et/ou en provoquant une thrombose. Cela mène souvent à de graves conséquences telles qu’un infarctus. Outre les problèmes liés à la sténose, les parois artérielles des régions criblées de plaque augmentent la rigidité des parois vasculaires, ce qui peut aggraver la condition du patient. Dans la population pédiatrique, la pathologie cardiovasculaire acquise la plus fréquente est la maladie de Kawasaki. Il s’agit d’une vasculite aigüe pouvant affecter l’intégrité structurale des parois des artères coronaires et mener à la formation d’anévrismes. Dans certains cas, ceux-ci entravent l’hémodynamie artérielle en engendrant une perfusion myocardique insuffisante et en activant la formation de thromboses. Le diagnostic de ces deux maladies coronariennes sont traditionnellement effectués à l’aide d’angiographies par fluoroscopie. Pendant ces examens paracliniques, plusieurs centaines de projections radiographiques sont acquises en séries suite à l’infusion artérielle d’un agent de contraste. Ces images révèlent la lumière des vaisseaux sanguins et la présence de lésions potentiellement pathologiques, s’il y a lieu. Parce que les séries acquises contiennent de l’information très dynamique en termes de mouvement du patient volontaire et involontaire (ex. battements cardiaques, respiration et déplacement d’organes), le clinicien base généralement son interprétation sur une seule image angiographique où des mesures géométriques sont effectuées manuellement ou semi-automatiquement par un technicien en radiologie. Bien que l’angiographie par fluoroscopie soit fréquemment utilisé partout dans le monde et souvent considéré comme l’outil de diagnostic “gold-standard” pour de nombreuses maladies vasculaires, la nature bidimensionnelle de cette modalité d’imagerie est malheureusement très limitante en termes de spécification géométrique des différentes régions pathologiques. En effet, la structure tridimensionnelle des sténoses et des anévrismes ne peut pas être pleinement appréciée en 2D car les caractéristiques observées varient selon la configuration angulaire de l’imageur. De plus, la présence de lésions affectant les artères coronaires peut ne pas refléter la véritable santé du myocarde, car des mécanismes compensatoires naturels (ex. vaisseaux----------ABSTRACT Cardiovascular disease continues to be the leading cause of death in North America. In adult and, alarmingly, ever younger populations, the so-called obesity epidemic largely driven by lifestyle factors that include poor diet, lack of exercise and smoking, incurs enormous stresses on the healthcare system. The primary cause of serious morbidity and mortality for these patients is atherosclerosis, the build up of plaque inside high pressure vessels like the coronary arteries. These lesions can lead to ischemic disease and may progress to precarious blood flow blockage or thrombosis, often with infarction or other severe consequences. Besides the stenosis-related outcomes, the arterial walls of plaque-ridden regions manifest increased stiffness, which may exacerbate negative patient prognosis. In pediatric populations, the most prevalent acquired cardiovascular pathology is Kawasaki disease. This acute vasculitis may affect the structural integrity of coronary artery walls and progress to aneurysmal lesions. These can hinder the blood flow’s hemodynamics, leading to inadequate downstream perfusion, and may activate thrombus formation which may lead to precarious prognosis. Diagnosing these two prominent coronary artery diseases is traditionally performed using fluoroscopic angiography. Several hundred serial x-ray projections are acquired during selective arterial infusion of a radiodense contrast agent, which reveals the vessels’ luminal area and possible pathological lesions. The acquired series contain highly dynamic information on voluntary and involuntary patient movement: respiration, organ displacement and heartbeat, for example. Current clinical analysis is largely limited to a single angiographic image where geometrical measures will be performed manually or semi-automatically by a radiological technician. Although widely used around the world and generally considered the gold-standard diagnosis tool for many vascular diseases, the two-dimensional nature of this imaging modality is limiting in terms of specifying the geometry of various pathological regions. Indeed, the 3D structures of stenotic or aneurysmal lesions may not be fully appreciated in 2D because their observable features are dependent on the angular configuration of the imaging gantry. Furthermore, the presence of lesions in the coronary arteries may not reflect the true health of the myocardium, as natural compensatory mechanisms may obviate the need for further intervention. In light of this, cardiac magnetic resonance perfusion imaging is increasingly gaining attention and clinical implementation, as it offers a direct assessment of myocardial tissue viability following infarction or suspected coronary artery disease. This type of modality is plagued, however, by motion similar to that present in fluoroscopic imaging. This issue predisposes clinicians to laborious manual intervention in order to align anatomical structures in sequential perfusion frames, thus hindering automation o

Automatic Spatiotemporal Analysis of Cardiac Image Series

RÉSUMÉ À ce jour, les maladies cardiovasculaires demeurent au premier rang des principales causes de décès en Amérique du Nord. Chez l’adulte et au sein de populations de plus en plus jeunes, la soi-disant épidémie d’obésité entraînée par certaines habitudes de vie tels que la mauvaise alimentation, le manque d’exercice et le tabagisme est lourde de conséquences pour les personnes affectées, mais aussi sur le système de santé. La principale cause de morbidité et de mortalité chez ces patients est l’athérosclérose, une accumulation de plaque à l’intérieur des vaisseaux sanguins à hautes pressions telles que les artères coronaires. Les lésions athérosclérotiques peuvent entraîner l’ischémie en bloquant la circulation sanguine et/ou en provoquant une thrombose. Cela mène souvent à de graves conséquences telles qu’un infarctus. Outre les problèmes liés à la sténose, les parois artérielles des régions criblées de plaque augmentent la rigidité des parois vasculaires, ce qui peut aggraver la condition du patient. Dans la population pédiatrique, la pathologie cardiovasculaire acquise la plus fréquente est la maladie de Kawasaki. Il s’agit d’une vasculite aigüe pouvant affecter l’intégrité structurale des parois des artères coronaires et mener à la formation d’anévrismes. Dans certains cas, ceux-ci entravent l’hémodynamie artérielle en engendrant une perfusion myocardique insuffisante et en activant la formation de thromboses. Le diagnostic de ces deux maladies coronariennes sont traditionnellement effectués à l’aide d’angiographies par fluoroscopie. Pendant ces examens paracliniques, plusieurs centaines de projections radiographiques sont acquises en séries suite à l’infusion artérielle d’un agent de contraste. Ces images révèlent la lumière des vaisseaux sanguins et la présence de lésions potentiellement pathologiques, s’il y a lieu. Parce que les séries acquises contiennent de l’information très dynamique en termes de mouvement du patient volontaire et involontaire (ex. battements cardiaques, respiration et déplacement d’organes), le clinicien base généralement son interprétation sur une seule image angiographique où des mesures géométriques sont effectuées manuellement ou semi-automatiquement par un technicien en radiologie. Bien que l’angiographie par fluoroscopie soit fréquemment utilisé partout dans le monde et souvent considéré comme l’outil de diagnostic “gold-standard” pour de nombreuses maladies vasculaires, la nature bidimensionnelle de cette modalité d’imagerie est malheureusement très limitante en termes de spécification géométrique des différentes régions pathologiques. En effet, la structure tridimensionnelle des sténoses et des anévrismes ne peut pas être pleinement appréciée en 2D car les caractéristiques observées varient selon la configuration angulaire de l’imageur. De plus, la présence de lésions affectant les artères coronaires peut ne pas refléter la véritable santé du myocarde, car des mécanismes compensatoires naturels (ex. vaisseaux----------ABSTRACT Cardiovascular disease continues to be the leading cause of death in North America. In adult and, alarmingly, ever younger populations, the so-called obesity epidemic largely driven by lifestyle factors that include poor diet, lack of exercise and smoking, incurs enormous stresses on the healthcare system. The primary cause of serious morbidity and mortality for these patients is atherosclerosis, the build up of plaque inside high pressure vessels like the coronary arteries. These lesions can lead to ischemic disease and may progress to precarious blood flow blockage or thrombosis, often with infarction or other severe consequences. Besides the stenosis-related outcomes, the arterial walls of plaque-ridden regions manifest increased stiffness, which may exacerbate negative patient prognosis. In pediatric populations, the most prevalent acquired cardiovascular pathology is Kawasaki disease. This acute vasculitis may affect the structural integrity of coronary artery walls and progress to aneurysmal lesions. These can hinder the blood flow’s hemodynamics, leading to inadequate downstream perfusion, and may activate thrombus formation which may lead to precarious prognosis. Diagnosing these two prominent coronary artery diseases is traditionally performed using fluoroscopic angiography. Several hundred serial x-ray projections are acquired during selective arterial infusion of a radiodense contrast agent, which reveals the vessels’ luminal area and possible pathological lesions. The acquired series contain highly dynamic information on voluntary and involuntary patient movement: respiration, organ displacement and heartbeat, for example. Current clinical analysis is largely limited to a single angiographic image where geometrical measures will be performed manually or semi-automatically by a radiological technician. Although widely used around the world and generally considered the gold-standard diagnosis tool for many vascular diseases, the two-dimensional nature of this imaging modality is limiting in terms of specifying the geometry of various pathological regions. Indeed, the 3D structures of stenotic or aneurysmal lesions may not be fully appreciated in 2D because their observable features are dependent on the angular configuration of the imaging gantry. Furthermore, the presence of lesions in the coronary arteries may not reflect the true health of the myocardium, as natural compensatory mechanisms may obviate the need for further intervention. In light of this, cardiac magnetic resonance perfusion imaging is increasingly gaining attention and clinical implementation, as it offers a direct assessment of myocardial tissue viability following infarction or suspected coronary artery disease. This type of modality is plagued, however, by motion similar to that present in fluoroscopic imaging. This issue predisposes clinicians to laborious manual intervention in order to align anatomical structures in sequential perfusion frames, thus hindering automation o