Towards Robot Autonomy in Medical Procedures Via Visual Localization and Motion Planning

Robots performing medical procedures with autonomous capabilities have the potential to positively effect patient care and healthcare system efficiency. These benefits can be realized by autonomous robots facilitating novel procedures, increasing operative efficiency, standardizing intra- and inter-physician performance, democratizing specialized care, and focusing the physician’s time on subtasks that best leverage their expertise. However, enabling medical robots to act autonomously in a procedural environment is extremely challenging. The deforming and unstructured nature of the environment, the lack of features in the anatomy, and sensor size constraints coupled with the millimeter level accuracy required for safe medical procedures introduce a host of challenges not faced by robots operating in structured environments such as factories or warehouses. Robot motion planning and localization are two fundamental abilities for enabling robot autonomy. Motion planning methods compute a sequence of safe and feasible motions for a robot to accomplish a specified task, where safe and feasible are defined by constraints with respect to the robot and its environment. Localization methods estimate the position and orientation of a robot in its environment. Developing such methods for medical robots that overcome the unique challenges in procedural environments is critical for enabling medical robot autonomy. In this dissertation, I developed and evaluated motion planning and localization algorithms towards robot autonomy in medical procedures. A majority of my work was done in the context of an autonomous medical robot built for enhanced lung nodule biopsy. First, I developed a dataset of medical environments spanning various organs and procedures to foster future research into medical robots and automation. I used this data in my own work described throughout this dissertation. Next, I used motion planning to characterize the capabilities of the lung nodule biopsy robot compared to existing clinical tools and I highlighted trade-offs in robot design considerations. Then, I conducted a study to experimentally demonstrate the benefits of the autonomous lung robot in accessing otherwise hard-to-reach lung nodules. I showed that the robot enables access to lung regions beyond the reach of existing clinical tools with millimeter-level accuracy sufficient for accessing the smallest clinically operable nodules. Next, I developed a localization method to estimate the bronchoscope’s position and orientation in the airways with respect to a preoperatively planned needle insertion pose. The method can be used by robotic bronchoscopy systems and by traditional manually navigated bronchoscopes. The method is designed to overcome challenges with tissue motion and visual homogeneity in the airways. I demonstrated the success of this method in simulated lungs undergoing respiratory motion and showed the method’s ability to generalize across patients.Doctor of Philosoph

Pulmonary Image Segmentation and Registration Algorithms: Towards Regional Evaluation of Obstructive Lung Disease

Pulmonary imaging, including pulmonary magnetic resonance imaging (MRI) and computed tomography (CT), provides a way to sensitively and regionally measure spatially heterogeneous lung structural-functional abnormalities. These unique imaging biomarkers offer the potential for better understanding pulmonary disease mechanisms, monitoring disease progression and response to therapy, and developing novel treatments for improved patient care. To generate these regional lung structure-function measurements and enable broad clinical applications of quantitative pulmonary MRI and CT biomarkers, as a first step, accurate, reproducible and rapid lung segmentation and registration methods are required. In this regard, we first developed a 1H MRI lung segmentation algorithm that employs complementary hyperpolarized 3He MRI functional information for improved lung segmentation. The 1H-3He MRI joint segmentation algorithm was formulated as a coupled continuous min-cut model and solved through convex relaxation, for which a dual coupled continuous max-flow model was proposed and a max-flow-based efficient numerical solver was developed. Experimental results on a clinical dataset of 25 chronic obstructive pulmonary disease (COPD) patients ranging in disease severity demonstrated that the algorithm provided rapid lung segmentation with high accuracy, reproducibility and diminished user interaction. We then developed a general 1H MRI left-right lung segmentation approach by exploring the left-to-right lung volume proportion prior. The challenging volume proportion-constrained multi-region segmentation problem was approximated through convex relaxation and equivalently represented by a max-flow model with bounded flow conservation conditions. This gave rise to a multiplier-based high performance numerical implementation based on convex optimization theories. In 20 patients with mild- to-moderate and severe asthma, the approach demonstrated high agreement with manual segmentation, excellent reproducibility and computational efficiency. Finally, we developed a CT-3He MRI deformable registration approach that coupled the complementary CT-1H MRI registration. The joint registration problem was solved by exploring optical-flow techniques, primal-dual analyses and convex optimization theories. In a diverse group of patients with asthma and COPD, the registration approach demonstrated lower target registration error than single registration and provided fast regional lung structure-function measurements that were strongly correlated with a reference method. Collectively, these lung segmentation and registration algorithms demonstrated accuracy, reproducibility and workflow efficiency that all may be clinically-acceptable. All of this is consistent with the need for broad and large-scale clinical applications of pulmonary MRI and CT

Coronary Artery Segmentation and Motion Modelling

Conventional coronary artery bypass surgery requires invasive sternotomy and the use of a cardiopulmonary bypass, which leads to long recovery period and has high infectious potential. Totally endoscopic coronary artery bypass (TECAB) surgery based on image guided robotic surgical approaches have been developed to allow the clinicians to conduct the bypass surgery off-pump with only three pin holes incisions in the chest cavity, through which two robotic arms and one stereo endoscopic camera are inserted. However, the restricted field of view of the stereo endoscopic images leads to possible vessel misidentification and coronary artery mis-localization. This results in 20-30% conversion rates from TECAB surgery to the conventional approach. We have constructed patient-specific 3D + time coronary artery and left ventricle motion models from preoperative 4D Computed Tomography Angiography (CTA) scans. Through temporally and spatially aligning this model with the intraoperative endoscopic views of the patient's beating heart, this work assists the surgeon to identify and locate the correct coronaries during the TECAB precedures. Thus this work has the prospect of reducing the conversion rate from TECAB to conventional coronary bypass procedures. This thesis mainly focus on designing segmentation and motion tracking methods of the coronary arteries in order to build pre-operative patient-specific motion models. Various vessel centreline extraction and lumen segmentation algorithms are presented, including intensity based approaches, geometric model matching method and morphology-based method. A probabilistic atlas of the coronary arteries is formed from a group of subjects to facilitate the vascular segmentation and registration procedures. Non-rigid registration framework based on a free-form deformation model and multi-level multi-channel large deformation diffeomorphic metric mapping are proposed to track the coronary motion. The methods are applied to 4D CTA images acquired from various groups of patients and quantitatively evaluated

Doctor of Philosophy

dissertationImage-based biomechanics, particularly numerical modeling using subject-specific data obtained via imaging, has proven useful for elucidating several biomechanical processes, such as prediction of deformation due to external loads, applicable to both normal function and pathophysiology of various organs. As the field evolves towards applications that stretch the limits of imaging hardware and acquisition time, the information traditionally expected as input for numerical routines often becomes incomplete or ambiguous, and requires specific acquisition and processing strategies to ensure physical accuracy and compatibility with predictive mathematical modeling. These strategies, often derivatives or specializations of traditional mechanics, effectively extend the nominal capability of medical imaging hardware providing subject-specific information coupled with the option of using the results for predictive numerical simulations. This research deals with the development of tools for extracting mechanical measurements from a finite set of imaging data and finite element analysis in the context of constructing structural atlases of the heart, understanding the biomechanics of the venous vasculature, and right ventricular failure. The tools include: (1) application of Hyperelastic Warping image registration to displacement-encoded MRI for reconstructing absolute displacement fields, (2) combination of imaging and a material parameter identification approach to measure morphology, deformation, and mechanical properties of vascular tissue, and (3) extrapolation of diffusion tensor MRI acquired at a single time point for the prediction the structural changes across the cardiac cycle with mechanical simulations. Selected tools were then applied to evaluate structural changes in a reversible animal model for right ventricular failure due to pressure overload

Investigation of Neonatal Pulmonary Structure and Function via Proton and Hyperpolarized Gas Magnetic Resonance Imaging

Magnetic resonance imaging (MRI) is a modality that utilizes the phenomenon of nuclear magnetic resonance (NMR) to yield tomographic images of the body. Proton (1H) MRI has historically been successful in soft tissues but has suffered in the lung due to a variety of technical challenges, such as the low proton-density, rapid T2* relaxation time of the lung parenchymal tissue, and inherent physiological motion in the chest. Recent developments in radial ultrashort echo time (UTE) MRI have in part overcome these issues. In addition, there has been much progress in techniques for hyperpolarization of noble gases (3He and 129Xe) out of thermal equilibrium via spin exchange optical pumping, which can greatly enhance the gas NMR signal such that it is detectable within the airspaces of the lung on MRI. The lung is a unique organ due to its complex structural and functional dynamics, and its early development through the neonatal (newborn) period is not yet well understood in normal or abnormal conditions. Pulmonary morbidities are relatively common in infants and are present in a majority of patients admitted to the neonatal intensive care unit, often stemming from preterm birth and/or congenital defects. Current clinical lung imaging in these patients is typically limited to chest x-ray radiography, which does not provide tomographic information and so has lowered sensitivity. More rarely, x-ray computed tomography (CT) is used but exposes infants to ionizing radiation and typically requires sedation, both of which pose increased risks to pediatric patients. Thus the opportunity is ripe for application of novel pulmonary MRI techniques to the infant population. However, MR imaging of very small pulmonary structure and microstructure requires fundamental changes in the imaging theory of both 1H UTE MRI and hyperpolarized gas diffusion MRI. Furthermore, such young patients are often non-compliant, yielding a need for new and innovative techniques for monitoring respiratory and bulk motion. This dissertation describes methodology development and provides experimental results in both 1H UTE MRI and hyperpolarized 3He and 129Xe gas diffusion MRI, with investigation into the structure and function of infant lungs at both the macrostructural and microstructural level. In particular, anisotropically restricted gas diffusion within infant alveolar microstructure is investigated as a measurement of airspace size and geometry. Additionally, the phenomenon of respiratory and bulk motion-tracking via modulation of the k-space center\u27s magnitude and phase is explored and applied via UTE MRI in various neonatal pulmonary conditions to extract imaging-based metrics of diagnostic value. Further, the proton-density regime of pulmonary UTE MRI is validated in translational applications. These techniques are applied in infants with various pulmonary conditions, including patients diagnosed with bronchopulmonary dysplasia, congenital diaphragmatic hernia, esophageal atresia/tracheoesophageal fistula, tracheomalacia, and no suspected lung disease. In addition, explanted lung specimens from both infants with and without lung disease are examined. Development and implementation of these techniques involves a strong understanding of the physics-based theory of NMR, hyperpolarization, and MR imaging, in addition to foundations in hardware, software, and image analysis techniques. This thesis first outlines the theory and background of NMR, MRI, and pulmonary physiology and development (Part I), then proceeds into the theory, equipment, and imaging experiments for hyperpolarized gas diffusion MRI in infant lung airspaces (Part II), and finally details the theory, data processing methods, and applications of pulmonary UTE MRI in infant patients (Part III). The potential for clinical translation of the neonatal pulmonary MRI methods presented in this dissertation is very high, with the foundations of these techniques firmly rooted in the laws of physics

Automatic segmentation of wall structures from cardiac images

One important topic in medical image analysis is segmenting wall structures from different cardiac medical imaging modalities such as computed tomography (CT) and magnetic resonance imaging (MRI). This task is typically done by radiologists either manually or semi-automatically, which is a very time-consuming process. To reduce the laborious human efforts, automatic methods have become popular in this research. In this thesis, features insensitive to data variations are explored to segment the ventricles from CT images and extract the left atrium from MR images. As applications, the segmentation results are used to facilitate cardiac disease analysis. Specifically, 1. An automatic method is proposed to extract the ventricles from CT images by integrating surface decomposition with contour evolution techniques. In particular, the ventricles are first identified on a surface extracted from patient-specific image data. Then, the contour evolution is employed to refine the identified ventricles. The proposed method is robust to variations of ventricle shapes, volume coverages, and image quality. 2. A variational region-growing method is proposed to segment the left atrium from MR images. Because of the localized property of this formulation, the proposed method is insensitive to data variabilities that are hard to handle by globalized methods. 3. In applications, a geometrical computational framework is proposed to estimate the myocardial mass at risk caused by stenoses. In addition, the segmentation of the left atrium is used to identify scars for MR images of post-ablation.PhDCommittee Chair: Yezzi, Anthony; Committee Co-Chair: Tannenbaum, Allen; Committee Member: Egerstedt, Magnus ; Committee Member: Fedele, Francesco ; Committee Member: Stillman, Arthur; Committee Member: Vela,Patrici

Computer integrated system: medical imaging & visualization

The intent of this book’s conception is to present research work using a user centered design approach. Due to space constraints, the story of the journey, included in this book is relatively brief. However we believe that it manages to adequately represent the story of the journey, from its humble beginnings in 2008 to the point where it visualizes future trends amongst both researchers and practitioners across the Computer Science and Medical disciplines. This book aims not only to present a representative sampling of real-world collaboration between said disciplines but also to provide insights into the different aspects related to the use of real-world Computer Assisted Medical applications. Readers and potential clients should find the information particularly useful in analyzing the benefits of collaboration between these two fields, the products in and of their institutions. The work discussed here is a compilation of the work of several PhD students under my supervision, who have since graduated and produced several publications either in journals or proceedings of conferences. As their work has been published, this book will be more focused on the research methodology based on medical technology used in their research. The research work presented in this book partially encompasses the work under the MOA for collaborative Research and Development in the field of Computer Assisted Surgery and Diagnostics pertaining to Thoracic and Cardiovascular Diseases between UPM, UKM and IJN, spanning five years beginning from 15 Feb 2013