Augmented Reality and Artificial Intelligence in Image-Guided and Robot-Assisted Interventions

In minimally invasive orthopedic procedures, the surgeon places wires, screws, and surgical implants through the muscles and bony structures under image guidance. These interventions require alignment of the pre- and intra-operative patient data, the intra-operative scanner, surgical instruments, and the patient. Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. State of the art approaches often support the surgeon by using external navigation systems or ill-conditioned image-based registration methods that both have certain drawbacks. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. Consequently, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. This dissertation investigates the applications of AR, artificial intelligence, and robotics in interventional medicine. Our solutions were applied in a broad spectrum of problems for various tasks, namely improving imaging and acquisition, image computing and analytics for registration and image understanding, and enhancing the interventional visualization. The benefits of these approaches were also discovered in robot-assisted interventions. We revealed how exemplary workflows are redefined via AR by taking full advantage of head-mounted displays when entirely co-registered with the imaging systems and the environment at all times. The proposed AR landscape is enabled by co-localizing the users and the imaging devices via the operating room environment and exploiting all involved frustums to move spatial information between different bodies. The system's awareness of the geometric and physical characteristics of X-ray imaging allows the exploration of different human-machine interfaces. We also leveraged the principles governing image formation and combined it with deep learning and RGBD sensing to fuse images and reconstruct interventional data. We hope that our holistic approaches towards improving the interface of surgery and enhancing the usability of interventional imaging, not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications

A Review on Advances in Intra-operative Imaging for Surgery and Therapy: Imagining the Operating Room of the Future

none4openZaffino, Paolo; Moccia, Sara; De Momi, Elena; Spadea, Maria FrancescaZaffino, Paolo; Moccia, Sara; De Momi, Elena; Spadea, Maria Francesc

CT from Motion: Volumetric Capture of Moving Shapes with X-rays and Videos

International audienceIn this paper, we consider the capture of dense volumetric X-ray attenuation models of non-rigidly moving samples. Traditional 3D medical imaging apparatus, e.g. CT or MRI, do not easily adapt to shapes that deform significantly such as a moving hand. We propose an approach that simultaneously recovers dense volumetric shape and motion information by combining video and X-ray modalities. Multiple colour images are captured to track shape surfaces while a single X-ray device is used to infer inner attenu-ations. The approach does not assume prior models which makes it versatile and easy to generalise over different shapes. Results on synthetic and real-life data are presented that demonstrate the approach feasibility with a limited number of X-ray views. The resulting dense 4D attenuation data provides unprecedented insights for motion analysis

Augmented Image-Guidance for Transcatheter Aortic Valve Implantation

The introduction of transcatheter aortic valve implantation (TAVI), an innovative stent-based technique for delivery of a bioprosthetic valve, has resulted in a paradigm shift in treatment options for elderly patients with aortic stenosis. While there have been major advancements in valve design and access routes, TAVI still relies largely on single-plane fluoroscopy for intraoperative navigation and guidance, which provides only gross imaging of anatomical structures. Inadequate imaging leading to suboptimal valve positioning contributes to many of the early complications experienced by TAVI patients, including valve embolism, coronary ostia obstruction, paravalvular leak, heart block, and secondary nephrotoxicity from contrast use. A potential method of providing improved image-guidance for TAVI is to combine the information derived from intra-operative fluoroscopy and TEE with pre-operative CT data. This would allow the 3D anatomy of the aortic root to be visualized along with real-time information about valve and prosthesis motion. The combined information can be visualized as a `merged\u27 image where the different imaging modalities are overlaid upon each other, or as an `augmented\u27 image, where the location of key target features identified on one image are displayed on a different imaging modality. This research develops image registration techniques to bring fluoroscopy, TEE, and CT models into a common coordinate frame with an image processing workflow that is compatible with the TAVI procedure. The techniques are designed to be fast enough to allow for real-time image fusion and visualization during the procedure, with an intra-procedural set-up requiring only a few minutes. TEE to fluoroscopy registration was achieved using a single-perspective TEE probe pose estimation technique. The alignment of CT and TEE images was achieved using custom-designed algorithms to extract aortic root contours from XPlane TEE images, and matching the shape of these contours to a CT-derived surface model. Registration accuracy was assessed on porcine and human images by identifying targets (such as guidewires or coronary ostia) on the different imaging modalities and measuring the correspondence of these targets after registration. The merged images demonstrated good visual alignment of aortic root structures, and quantitative assessment measured an accuracy of less than 1.5mm error for TEE-fluoroscopy registration and less than 6mm error for CT-TEE registration. These results suggest that the image processing techniques presented have potential for development into a clinical tool to guide TAVI. Such a tool could potentially reduce TAVI complications, reducing morbidity and mortality and allowing for a safer procedure

Contributions to the improvement of image quality in CBCT and CBμCT and application in the development of a CBμCT system

During the last years cone-beam x-ray CT (CBCT) has been established as a widespread imaging technique and a feasible alternative to conventional CT for dedicated imaging tasks for which the limited flexibility offered by conventional CT advises the development of dedicated designs. CBCT systems are starting to be routinely used in image guided radiotherapy; image guided surgery using C-arms; scan of body parts such as the sinuses, the breast or extremities; and, especially, in preclinical small-animal imaging, often coupled to molecular imaging systems. Despite the research efforts advocated to the advance of CBCT, the challenges introduced by the use of large cone angles and two-dimensional detectors are a field of vigorous research towards the improvement of CBCT image quality. Moreover, systems for small-animal imaging add to the challenges posed by clinical CBCT the need of higher resolution to obtain equivalent image quality in much smaller subjects. This thesis contributes to the progress of CBCT imaging by addressing a variety of issues affecting image quality in CBCT in general and in CBCT for small-animal imaging (CBμCT). As part of this work we have assessed and optimized the performance of CBμCT systems for different imaging tasks. To this end, we have developed a new CBμCT system with variable geometry and all the required software tools for acquisition, calibration and reconstruction. The system served as a tool for the optimization of the imaging process and for the study of image degradation effects in CBμCT, as well as a platform for biological research using small animals. The set of tools for the accurate study of CBCT was completed by developing a fast Monte Carlo simulation engine based on GPUs, specifically devoted to the realistic estimation of scatter and its effects on image quality in arbitrary CBCT configurations, with arbitrary spectra, detector response, and antiscatter grids. This new Monte Carlo engine outperformed current simulation platforms by more than an order of magnitude. Due to the limited options for simulation of spectra in microfocus x-ray sources used in CBμCT, we contributed in this thesis a new spectra generation model based on an empirical model for conventional radiology and mammography sources modified in accordance to experimental data. The new spectral model showed good agreement with experimental exposure and attenuation data for different materials. The developed tools for CBμCT research were used for the study of detector performance in terms of dynamic range. The dynamic range of the detector was characterized together with its effect on image quality. As a result, a new simple method for the extension of the dynamic range of flat-panel detectors was proposed and evaluated. The method is based on a modified acquisition process and a mathematical treatment of the acquired data. Scatter is usually identified as one of the major causes of image quality degradation in CBCT. For this reason the developed Monte Carlo engine was applied to the in-depth study of the effects of scatter for a representative range of CBCT embodiments used in the clinical and preclinical practice. We estimated the amount and spatial distribution of the total scatter fluence and the individual components within. The effect of antiscatter grids in improving image quality and in noise was also evaluated. We found a close relation between scatter and the air gap of the system, in line with previous results in the literature. We also observed a non-negligible contribution of forward-directed scatter that is responsible to a great extent for streak artifacts in CBCT. The spatial distribution of scatter was significantly affected by forward scatter, somewhat challenging the usual assumption that the scatter distribution mostly contains low-frequencies. Antiscatter grids showed to be effective for the reduction of cupping, but they showed a much lower performance when dealing with streaks and a shift toward high frequencies of the scatter distributions. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------A lo largo de los últimos años, el TAC de rayos X de haz cónico (CBCT, de “conebeam” CT) se ha posicionado como una de las técnicas de imagen más ampliamente usadas. El CBCT se ha convertido en una alternativa factible al TAC convencional en tareas de imagen específicas para las que la flexibilidad limitada ofrecida por este hace recomendable el desarrollo de sistemas de imagen dedicados. De esta forma, el CBCT está empezando a usarse de forma rutinaria en varios campos entre los que se incluyen la radioterapia guiada por imagen, la cirugía guiada por imagen usando arcos en C, imagen de partes de la anatomía en las que el TAC convencional no es apropiado, como los senos nasales, las extremidades o la mama, y, especialmente el campo de imagen preclínica con pequeño animal. Los sistemas CBCT usados en este último campo se encuentran habitualmente combinados con sistemas de imagen molecular. A pesar del trabajo de investigación dedicado al avance de la técnica CBCT en los últimos años, los retos introducidos por el uso de haces cónicos y de detectores bidimensionales son un campo candente para la investigación médica, con el objetivo de obtener una calidad de imagen equivalente o superior a la proporcionada por el TAC convencional. En el caso de imagen preclínica, a los retos generados por el uso de CBCT se une la necesidad de una mayor resolución de imagen que permita observar estructuras anatómicas con el mismo nivel de detalle obtenido para humanos. Esta tesis contribuye al progreso del CBCT mediante el estudio de usa serie de efectos que afectan a la calidad de imagen de CBCT en general y en el ámbito preclínico en particular. Como parte de este trabajo, hemos evaluado y optimizado el rendimiento de sistemas CBCT preclínicos en función de la tarea de imagen concreta. Con este fin se ha desarrollado un sistema CBCT para pequeños animales con geometría variable y todas las herramientas necesarias para la adquisición, calibración y reconstrucción de imagen. El sistema sirve como base para la optimización de protocolos de adquisición y para el estudio de fuentes de degradación de imagen además de constituir una plataforma para la investigación biológica en pequeño animal. El conjunto de herramientas para el estudio del CBCT se completó con el desarrollo de una plataforma acelerada de simulación Monte Carlo basada en GPUs, optimizada para la estimación de radiación dispersa en CBCT y sus efectos en la calidad de imagen. La plataforma desarrollada supera el rendimiento de las actuales en más de un orden de magnitud y permite la inclusión de espectros policromáticos de rayos X, de la respuesta realista del detector y de rejillas antiscatter. Debido a las escasas opciones ofrecidas por la literatura para la estimación de espectros de rayos X para fuentes microfoco usadas en imagen preclínica, en esta tesis se incluye el desarrollo de un nuevo modelo de generación de espectros, basado en un modelo existente para fuentes usadas en radiología y mamografía. El modelo fue modificado a partir de datos experimentales. La precisión del modelo presentado se comprobó mediante datos experimentales de exposición y atenuación para varios materiales. Las herramientas desarrolladas se usaron para estudiar el rendimiento de detectores de rayos tipo flat-panel en términos de rango dinámico, explorando los límites impuestos por el mismo en la calidad de imagen. Como resultado se propuso y evaluó un método para la extensión del rango dinámico de este tipo de detectores. El método se basa en la modificación del proceso de adquisición de imagen y en una etapa de postproceso de los datos adquiridos. El simulador Monte Carlo se empleó para el estudio detallado de la naturaleza, distribución espacial y efectos de la radiación dispersa en un rango de sistemas CBCT que cubre el espectro de aplicaciones propuestas en el entorno clínico y preclínico. Durante el estudio se inspeccionó la cantidad y distribución espacial de radiación dispersa y de sus componentes individuales y el efecto causado por la inclusión de rejillas antiscatter en términos de mejora de calidad de imagen y de ruido en la imagen. La distribución de radiación dispersa mostró una acentuada relación con la distancia entre muestra y detector en el equipo, en línea con resultados publicados previamente por otros autores. También se encontró una influencia no despreciable de componentes de radiación dispersa con bajos ángulos de desviación, poniendo en tela de juicio la tradicional asunción que considera que la distribución espacial de la radiación dispersa está formada casi exclusivamente por componentes de muy baja frecuencia. Las rejillas antiscatter demostraron ser efectivas para la reducción del artefacto de cupping, pero su efectividad para tratar artefactos en forma de línea (principalmente formados por radiación dispersa con bajo ángulo de desviación) resultó mucho menor. La inclusión de estas rejillas también enfatiza las componentes de alta frecuencia de la distribución espacial de la radiación dispersa

Investigation and Validation of Imaging Techniques for Mitral Valve Disease Diagnosis and Intervention

Mitral Valve Disease (MVD) describes a variety of pathologies that result in regurgitation of blood during the systolic phase of the cardiac cycle. Decisions in valvular disease management rely heavily on non-invasive imaging. Transesophageal echocardiography (TEE) is widely recognized as the key evaluation technique where backflow of high velocity blood can be visualized under Doppler. In most cases, TEE imaging is adequate for identifying mitral valve pathology, though the modality is often limited from signal dropout, artifacts and a restricted field of view. Quantitative analysis is an integral part of the overall assessment of valve morphology and gives objective evidence for both classification and guiding intervention of regurgitation. In addition, patient-specific models derived from diagnostic TEE images allow clinicians to gain insight into uniquely intricate anatomy prior to surgery. However, the heavy reliance on TEE segmentation for diagnosis and modelling has necessitated an evaluation of the accuracy of the oft-used mitral valve imaging modality. Dynamic cardiac 4D-Computed Tomography (4D-CT) is emerging as a valuable tool for diagnosis, quantification and assessment of cardiac diseases. This modality has the potential to provide a high quality rendering of the mitral valve and subvalvular apparatus, to provide a more complete picture of the underlying morphology. However, application of dynamic CT to mitral valve imaging is especially challenging due to the large and rapid motion of the valve leaflets. It is therefore necessary to investigate the accuracy and level of precision by which dynamic CT captures mitral valve motion throughout the cardiac cycle. To do this, we design and construct a silicone and bovine quasi-static mitral valve phantom which can simulate a range of ECG-gated heart rates and reproduce physiologic valve motion over the cardiac cycle. In this study, we discovered that the dynamic CT accurately captures the underlying valve movement, but with a higher prevalence of image artifacts as leaflet and chordae motion increases due to elevated heart rates. In a subsequent study, we acquire simultaneous CT and TEE images of both a silicone mitral valve phantom and an iodine-stained bovine mitral valve. We propose a pipeline to use CT as the ground truth to study the relationship between TEE intensities and the underlying valve morphology. Preliminary results demonstrate that with an optimized threshold selection based solely on TEE pixel intensities, only 40\% of pixels are correctly classified as part of the valve. In addition, we have shown that emphasizing the centre-line rather than the boundaries of high intensity TEE image regions provides a better representation and segmentation of the valve morphology. This work has the potential to inform and augment the use of TEE for diagnosis and modelling of the mitral valve in the clinical workflow for MVD