Virtual and Augmented Reality Techniques for Minimally Invasive Cardiac Interventions: Concept, Design, Evaluation and Pre-clinical Implementation

While less invasive techniques have been employed for some procedures, most intracardiac interventions are still performed under cardiopulmonary bypass, on the drained, arrested heart. The progress toward off-pump intracardiac interventions has been hampered by the lack of adequate visualization inside the beating heart. This thesis describes the development, assessment, and pre-clinical implementation of a mixed reality environment that integrates pre-operative imaging and modeling with surgical tracking technologies and real-time ultrasound imaging. The intra-operative echo images are augmented with pre-operative representations of the cardiac anatomy and virtual models of the delivery instruments tracked in real time using magnetic tracking technologies. As a result, the otherwise context-less images can now be interpreted within the anatomical context provided by the anatomical models. The virtual models assist the user with the tool-to-target navigation, while real-time ultrasound ensures accurate positioning of the tool on target, providing the surgeon with sufficient information to ``see\u27\u27 and manipulate instruments in absence of direct vision. Several pre-clinical acute evaluation studies have been conducted in vivo on swine models to assess the feasibility of the proposed environment in a clinical context. Following direct access inside the beating heart using the UCI, the proposed mixed reality environment was used to provide the necessary visualization and navigation to position a prosthetic mitral valve on the the native annulus, or to place a repair patch on a created septal defect in vivo in porcine models. Following further development and seamless integration into the clinical workflow, we hope that the proposed mixed reality guidance environment may become a significant milestone toward enabling minimally invasive therapy on the beating heart

Dynamic Image Processing for Guidance of Off-pump Beating Heart Mitral Valve Repair

Compared to conventional open heart procedures, minimally invasive off-pump beating heart mitral valve repair aims to deliver equivalent treatment for mitral regurgitation with reduced trauma and side effects. However, minimally invasive approaches are often limited by the lack of a direct view to surgical targets and/or tools, a challenge that is compounded by potential movement of the target during the cardiac cycle. For this reason, sophisticated image guidance systems are required in achieving procedural efficiency and therapeutic success. The development of such guidance systems is associated with many challenges. For example, the system should be able to provide high quality visualization of both cardiac anatomy and motion, as well as augmenting it with virtual models of tracked tools and targets. It should have the capability of integrating pre-operative images to the intra-operative scenario through registration techniques. The computation speed must be sufficiently fast to capture the rapid cardiac motion. Meanwhile, the system should be cost effective and easily integrated into standard clinical workflow. This thesis develops image processing techniques to address these challenges, aiming to achieve a safe and efficient guidance system for off-pump beating heart mitral valve repair. These techniques can be divided into two categories, using 3D and 2D image data respectively. When 3D images are accessible, a rapid multi-modal registration approach is proposed to link the pre-operative CT images to the intra-operative ultrasound images. The ultrasound images are used to display the real time cardiac motion, enhanced by CT data serving as high quality 3D context with annotated features. I also developed a method to generate synthetic dynamic CT images, aiming to replace real dynamic CT data in such a guidance system to reduce the radiation dose applied to the patients. When only 2D images are available, an approach is developed to track the feature of interest, i.e. the mitral annulus, based on bi-plane ultrasound images and a magnetic tracking system. The concept of modern GPU-based parallel computing is employed in most of these approaches to accelerate the computation in order to capture the rapid cardiac motion with desired accuracy. Validation experiments were performed on phantom, animal and human data. The overall accuracy of registration and feature tracking with respect to the mitral annulus was about 2-3mm with computation time of 60-400ms per frame, sufficient for one update per cardiac cycle. It was also demonstrated in the results that the synthetic CT images can provide very similar anatomical representations and registration accuracy compared to that of the real dynamic CT images. These results suggest that the approaches developed in the thesis have good potential for a safer and more effective guidance system for off-pump beating heart mitral valve repair

The Role of Visualization, Force Feedback, and Augmented Reality in Minimally Invasive Heart Valve Repair

New cardiovascular techniques have been developed to address the unique requirements of high risk, elderly, surgical patients with heart valve disease by avoiding both sternotomy and cardiopulmonary bypass. However, these technologies pose new challenges in visualization, force application, and intracardiac navigation. Force feedback and augmented reality (AR) can be applied to minimally invasive mitral valve repair and transcatheter aortic valve implantation (TAVI) techniques to potentially surmount these challenges. Our study demonstrated shorter operative times with three dimensional (3D) visualization compared to two dimensional (2D) visualization; however, both experts and novices applied significantly more force to cardiac tissue during 3D robotics-assisted mitral valve annuloplasty than during conventional open mitral valve annuloplasty. This finding suggests that 3D visualization does not fully compensate for the absence of haptic feedback in robotics-assisted cardiac surgery. Subsequently, using an innovative robotics-assisted surgical system design, we determined that direct haptic feedback may improve both expert and trainee performance using robotics-assisted techniques. We determined that during robotics-assisted mitral valve annuloplasty the use of either visual or direct force feedback resulted in a significant decrease in forces applied to cardiac tissue when compared to robotics-assisted mitral valve annuloplasty without force feedback. We presented NeoNav, an AR-enhanced echocardiograpy intracardiac guidance system for NeoChord off-pump mitral valve repair. Our study demonstrated superior tool navigation accuracy, significantly shorter navigation times, and reduced potential for injury with AR enhanced intracardiac navigation for off-pump transapical mitral valve repair with neochordae implantation. In addition, we applied the NeoNav system as a safe and inexpensive alternative imaging modality for TAVI guidance. We found that our proposed AR guidance system may achieve similar or better results than the current standard of care, contrast enhanced fluoroscopy, while eliminating the use of nephrotoxic contrast and ionizing radiation. These results suggest that the addition of both force feedback and augmented reality image guidance can improve both surgical performance and safety during minimally invasive robotics assisted and beating heart valve surgery, respectively

The role of imaging to support catheter ablation of atrial fibrillation

AbstractAtrial fibrillation (AF) ablation is a complex procedure that requires transseptal puncture and extensive manipulation with catheter(s) in the left atrium and pulmonary veins. Individual anatomy of these structures contributes to a challenge of AF ablation. The proximity of surrounding structures, such as esophagus, further increases risk of complications of this procedure. Increased risk of intracardiac thrombosis associated with AF is another factor that may complicate management of these patients. For all these reasons, imaging techniques play increasingly important role. Preprocedural imaging becomes important not only to rule out thrombus but also for assessment of anatomy of the PVs and left atrium, left atrial size and the extent of a substrate. Various forms of imaging help significantly during the procedure both with identification of anatomy and with catheter navigation. Many studies have shown increased efficacy, safety and decreased fluoroscopy times. After the procedure, imaging techniques such as echocardiography, CT or MR imaging are useful to diagnose potential complications. This paper briefly reviews clinical utility of different imaging tools for ablation of AF

Exploiting Temporal Image Information in Minimally Invasive Surgery

Minimally invasive procedures rely on medical imaging instead of the surgeons direct vision. While preoperative images can be used for surgical planning and navigation, once the surgeon arrives at the target site real-time intraoperative imaging is needed. However, acquiring and interpreting these images can be challenging and much of the rich temporal information present in these images is not visible. The goal of this thesis is to improve image guidance for minimally invasive surgery in two main areas. First, by showing how high-quality ultrasound video can be obtained by integrating an ultrasound transducer directly into delivery devices for beating heart valve surgery. Secondly, by extracting hidden temporal information through video processing methods to help the surgeon localize important anatomical structures. Prototypes of delivery tools, with integrated ultrasound imaging, were developed for both transcatheter aortic valve implantation and mitral valve repair. These tools provided an on-site view that shows the tool-tissue interactions during valve repair. Additionally, augmented reality environments were used to add more anatomical context that aids in navigation and in interpreting the on-site video. Other procedures can be improved by extracting hidden temporal information from the intraoperative video. In ultrasound guided epidural injections, dural pulsation provides a cue in finding a clear trajectory to the epidural space. By processing the video using extended Kalman filtering, subtle pulsations were automatically detected and visualized in real-time. A statistical framework for analyzing periodicity was developed based on dynamic linear modelling. In addition to detecting dural pulsation in lumbar spine ultrasound, this approach was used to image tissue perfusion in natural video and generate ventilation maps from free-breathing magnetic resonance imaging. A second statistical method, based on spectral analysis of pixel intensity values, allowed blood flow to be detected directly from high-frequency B-mode ultrasound video. Finally, pulsatile cues in endoscopic video were enhanced through Eulerian video magnification to help localize critical vasculature. This approach shows particular promise in identifying the basilar artery in endoscopic third ventriculostomy and the prostatic artery in nerve-sparing prostatectomy. A real-time implementation was developed which processed full-resolution stereoscopic video on the da Vinci Surgical System

Intracardiac Ultrasound Guided Systems for Transcatheter Cardiac Interventions

Transcatheter cardiac interventions are characterized by their percutaneous nature, increased patient safety, and low hospitalization times. Transcatheter procedures involve two major stages: navigation towards the target site and the positioning of tools to deliver the therapy, during which the interventionalists face the challenge of visualizing the anatomy and the relative position of the tools such as a guidewire. Fluoroscopic and transesophageal ultrasound (TEE) imaging are the most used techniques in cardiac procedures; however, they possess the disadvantage of radiation exposure and suboptimal imaging. This work explores the potential of intracardiac ultrasound (ICE) within an image guidance system (IGS) to facilitate the two stages of cardiac interventions. First, a novel 2.5D side-firing, conical Foresight ICE probe (Conavi Medical Inc., Toronto) is characterized, calibrated, and tracked using an electromagnetic sensor. The results indicate an acceptable tracking accuracy within some limitations. Next, an IGS is developed for navigating the vessels without fluoroscopy. A forward-looking, tracked ICE probe is used to reconstruct the vessel on a phantom which mimics the ultrasound imaging of an animal vena cava. Deep learning methods are employed to segment the complex vessel geometry from ICE imaging for the first time. The ICE-reconstructed vessel showed a clinically acceptable range of accuracy. Finally, a guidance system was developed to facilitate the positioning of tools during a tricuspid valve repair. The designed system potentially facilitates the positioning of the TriClip at the coaptation gap by pre-mapping the corresponding site of regurgitation in 3D tracking space

INTERFACE DESIGN FOR A VIRTUAL REALITY-ENHANCED IMAGE-GUIDED SURGERY PLATFORM USING SURGEON-CONTROLLED VIEWING TECHNIQUES

Initiative has been taken to develop a VR-guided cardiac interface that will display and deliver information without affecting the surgeons’ natural workflow while yielding better accuracy and task completion time than the existing setup. This paper discusses the design process, the development of comparable user interface prototypes as well as an evaluation methodology that can measure user performance and workload for each of the suggested display concepts. User-based studies and expert recommendations are used in conjunction to es­ tablish design guidelines for our VR-guided surgical platform. As a result, a better understanding of autonomous view control, depth display, and use of virtual context, is attained. In addition, three proposed interfaces have been developed to allow a surgeon to control the view of the virtual environment intra-operatively. Comparative evaluation of the three implemented interface prototypes in a simulated surgical task scenario, revealed performance advantages for stereoscopic and monoscopic biplanar display conditions, as well as the differences between three types of control modalities. One particular interface prototype demonstrated significant improvement in task performance. Design recommendations are made for this interface as well as the others as we prepare for prospective development iterations

Augmented Reality Ultrasound Guidance in Anesthesiology

Real-time ultrasound has become a mainstay in many image-guided interventions and increasingly popular in several percutaneous procedures in anesthesiology. One of the main constraints of ultrasound-guided needle interventions is identifying and distinguishing the needle tip from needle shaft in the image. Augmented reality (AR) environments have been employed to address challenges surrounding surgical tool visualization, navigation, and positioning in many image-guided interventions. The motivation behind this work was to explore the feasibility and utility of such visualization techniques in anesthesiology to address some of the specific limitations of ultrasound-guided needle interventions. This thesis brings together the goals, guidelines, and best development practices of functional AR ultrasound image guidance (AR-UIG) systems, examines the general structure of such systems suitable for applications in anesthesiology, and provides a series of recommendations for their development. The main components of such systems, including ultrasound calibration and system interface design, as well as applications of AR-UIG systems for quantitative skill assessment, were also examined in this thesis. The effects of ultrasound image reconstruction techniques, as well as phantom material and geometry on ultrasound calibration, were investigated. Ultrasound calibration error was reduced by 10% with synthetic transmit aperture imaging compared with B-mode ultrasound. Phantom properties were shown to have a significant effect on calibration error, which is a variable based on ultrasound beamforming techniques. This finding has the potential to alter how calibration phantoms are designed cognizant of the ultrasound imaging technique. Performance of an AR-UIG guidance system tailored to central line insertions was evaluated in novice and expert user studies. While the system outperformed ultrasound-only guidance with novice users, it did not significantly affect the performance of experienced operators. Although the extensive experience of the users with ultrasound may have affected the results, certain aspects of the AR-UIG system contributed to the lackluster outcomes, which were analyzed via a thorough critique of the design decisions. The application of an AR-UIG system in quantitative skill assessment was investigated, and the first quantitative analysis of needle tip localization error in ultrasound in a simulated central line procedure, performed by experienced operators, is presented. Most participants did not closely follow the needle tip in ultrasound, resulting in 42% unsuccessful needle placements and a 33% complication rate. Compared to successful trials, unsuccessful procedures featured a significantly greater (p=0.04) needle-tip to image-plane distance. Professional experience with ultrasound does not necessarily lead to expert level performance. Along with deliberate practice, quantitative skill assessment may reinforce clinical best practices in ultrasound-guided needle insertions. Based on the development guidelines, an AR-UIG system was developed to address the challenges in ultrasound-guided epidural injections. For improved needle positioning, this system integrated A-mode ultrasound signal obtained from a transducer housed at the tip of the needle. Improved needle navigation was achieved via enhanced visualization of the needle in an AR environment, in which B-mode and A-mode ultrasound data were incorporated. The technical feasibility of the AR-UIG system was evaluated in a preliminary user study. The results suggested that the AR-UIG system has the potential to outperform ultrasound-only guidance

A Platform for Robot-Assisted Intracardiac Catheter Navigation

Steerable catheters are routinely deployed in the treatment of cardiac arrhythmias. During invasive electrophysiology studies, the catheter handle is manipulated by an interventionalist to guide the catheter's distal section toward endocardium for pacing and ablation. Catheter manipulation requires dexterity and experience, and exposes the interventionalist to ionizing radiation. Through the course of this research, a platform was developed to assist and enhance the navigation of the catheter inside the cardiac chambers. This robotic platform replaces the interventionalist's hand in catheter manipulation and provides the option to force the catheter tip in arbitrary directions using a 3D input device or to automatically navigate the catheter to desired positions within a cardiac chamber by commanding the software to do so. To accomplish catheter navigation, the catheter was modeled as a continuum manipulator, and utilizing robot kinematics, catheter tip position control was designed and implemented. An electromagnetic tracking system was utilized to measure the position and orientation of two key points in catheter model, for position feedback to the control system. A software platform was developed to implement the navigation and control strategies and to interface with the robot, the 3D input device and the tracking system. The catheter modeling was validated through in-vitro experiments with a static phantom, and in-vivo experiments on three live swines. The feasibility of automatic navigation was also veri ed by navigating to three landmarks in the beating heart of swine subjects, and comparing their performance with that of an experienced interventionalist using quasi biplane fluoroscopy. The platform realizes automatic, assisted, and motorized navigation under the interventionalist's control, thus reducing the dependence of successful navigation on the dexterity and manipulation skills of the interventionalist, and providing a means to reduce the exposure to X-ray radiation. Upon further development, the platform could be adopted for human deployment

Insights into 3D printing in medical applications

Three-dimensional (3D) printing has been increasingly used in the medical field with reported applications showing great value in assisting clinical decision-making and improving patient care (1-10). Patient-specific 3D printed models derived from medical imaging datasets, mainly from computed tomography (CT) and magnetic resonance imaging (MRI) are shown to play an important role in pre-surgical planning and simulation of complex surgical procedures, medical education and patient-doctor communication (1-14)