Augmented Reality and Artificial Intelligence in Image-Guided and Robot-Assisted Interventions

In minimally invasive orthopedic procedures, the surgeon places wires, screws, and surgical implants through the muscles and bony structures under image guidance. These interventions require alignment of the pre- and intra-operative patient data, the intra-operative scanner, surgical instruments, and the patient. Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. State of the art approaches often support the surgeon by using external navigation systems or ill-conditioned image-based registration methods that both have certain drawbacks. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. Consequently, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. This dissertation investigates the applications of AR, artificial intelligence, and robotics in interventional medicine. Our solutions were applied in a broad spectrum of problems for various tasks, namely improving imaging and acquisition, image computing and analytics for registration and image understanding, and enhancing the interventional visualization. The benefits of these approaches were also discovered in robot-assisted interventions. We revealed how exemplary workflows are redefined via AR by taking full advantage of head-mounted displays when entirely co-registered with the imaging systems and the environment at all times. The proposed AR landscape is enabled by co-localizing the users and the imaging devices via the operating room environment and exploiting all involved frustums to move spatial information between different bodies. The system's awareness of the geometric and physical characteristics of X-ray imaging allows the exploration of different human-machine interfaces. We also leveraged the principles governing image formation and combined it with deep learning and RGBD sensing to fuse images and reconstruct interventional data. We hope that our holistic approaches towards improving the interface of surgery and enhancing the usability of interventional imaging, not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications

Three-dimensional modeling of the human jaw/teeth using optics and statistics.

Object modeling is a fundamental problem in engineering, involving talents from computer-aided design, computational geometry, computer vision and advanced manufacturing. The process of object modeling takes three stages: sensing, representation, and analysis. Various sensors may be used to capture information about objects; optical cameras and laser scanners are common with rigid objects, while X-ray, CT and MRI are common with biological organs. These sensors may provide a direct or an indirect inference about the object, requiring a geometric representation in the computer that is suitable for subsequent usage. Geometric representations that are compact, i.e., capture the main features of the objects with a minimal number of data points or vertices, fall into the domain of computational geometry. Once a compact object representation is in the computer, various analysis steps can be conducted, including recognition, coding, transmission, etc. The subject matter of this dissertation is object reconstruction from a sequence of optical images using shape from shading (SFS) and SFS with shape priors. The application domain is dentistry. Most of the SFS approaches focus on the computational part of the SFS problem, i.e. the numerical solution. As a result, the imaging model in most conventional SFS algorithms has been simplified under three simple, but restrictive assumptions: (1) the camera performs an orthographic projection of the scene, (2) the surface has a Lambertian reflectance and (3) the light source is a single point source at infinity. Unfortunately, such assumptions are no longer held in the case of reconstruction of real objects as intra-oral imaging environment for human teeth. In this work, we introduce a more realistic formulation of the SFS problem by considering the image formation components: the camera, the light source, and the surface reflectance. This dissertation proposes a non-Lambertian SFS algorithm under perspective projection which benefits from camera calibration parameters. The attenuation of illumination is taken account due to near-field imaging. The surface reflectance is modeled using the Oren-Nayar-Wolff model which accounts for the retro-reflection case. In this context, a new variational formulation is proposed that relates an evolving surface model with image information, taking into consideration that the image is taken by a perspective camera with known parameters. A new energy functional is formulated to incorporate brightness, smoothness and integrability constraints. In addition, to further improve the accuracy and practicality of the results, 3D shape priors are incorporated in the proposed SFS formulation. This strategy is motivated by the fact that humans rely on strong prior information about the 3D world around us in order to perceive 3D shape information. Such information is statistically extracted from training 3D models of the human teeth. The proposed SFS algorithms have been used in two different frameworks in this dissertation: a) holistic, which stitches a sequence of images in order to cover the entire jaw, and then apply the SFS, and b) piece-wise, which focuses on a specific tooth or a segment of the human jaw, and applies SFS using physical teeth illumination characteristics. To augment the visible portion, and in order to have the entire jaw reconstructed without the use of CT or MRI or even X-rays, prior information were added which gathered from a database of human jaws. This database has been constructed from an adult population with variations in teeth size, degradation and alignments. The database contains both shape and albedo information for the population. Using this database, a novel statistical shape from shading (SSFS) approach has been created. Extending the work on human teeth analysis, Finite Element Analysis (FEA) is adapted for analyzing and calculating stresses and strains of dental structures. Previous Finite Element (FE) studies used approximate 2D models. In this dissertation, an accurate three-dimensional CAD model is proposed. 3D stress and displacements of different teeth type are successfully carried out. A newly developed open-source finite element solver, Finite Elements for Biomechanics (FEBio), has been used. The limitations of the experimental and analytical approaches used for stress and displacement analysis are overcome by using FEA tool benefits such as dealing with complex geometry and complex loading conditions

Recent trends, technical concepts and components of computer-assisted orthopedic surgery systems: A comprehensive review

Computer-assisted orthopedic surgery (CAOS) systems have become one of the most important and challenging types of system in clinical orthopedics, as they enable precise treatment of musculoskeletal diseases, employing modern clinical navigation systems and surgical tools. This paper brings a comprehensive review of recent trends and possibilities of CAOS systems. There are three types of the surgical planning systems, including: systems based on the volumetric images (computer tomography (CT), magnetic resonance imaging (MRI) or ultrasound images), further systems utilize either 2D or 3D fluoroscopic images, and the last one utilizes the kinetic information about the joints and morphological information about the target bones. This complex review is focused on three fundamental aspects of CAOS systems: their essential components, types of CAOS systems, and mechanical tools used in CAOS systems. In this review, we also outline the possibilities for using ultrasound computer-assisted orthopedic surgery (UCAOS) systems as an alternative to conventionally used CAOS systems.Web of Science1923art. no. 519

A Review on Advances in Intra-operative Imaging for Surgery and Therapy: Imagining the Operating Room of the Future

none4openZaffino, Paolo; Moccia, Sara; De Momi, Elena; Spadea, Maria FrancescaZaffino, Paolo; Moccia, Sara; De Momi, Elena; Spadea, Maria Francesc

Assessment of Wear in Total Knee Arthroplasty Using Advanced Radiographic Techniques

Total knee arthroplasty (TKA) has become the gold standard approach for treating advanced osteoarthritis of the knee. Although the surgery continues to be very successful at relieving pain and restoring joint function, its longevity is challenged by wear and loosening of the implant components. This requires the patient to undergo a revision surgery to replace the implant, a much more challenging operation than primary arthroplasty. Wear of the polyethylene tibial inserts from TKA is assessed in vitro using mechanical wear simulator testing and by examining failed implants retrieved from patients during revision surgery, as well as with direct in vivo measurements. Current in vitro measurement tools provide only a global estimate of wear (failing to describe whether the wear has occurred on the articulating or backside surfaces, or stabilizing post), or are qualitative measurements, or lack resolution. Current in vivo measurement techniques are performed statically or quasi-statically, leading to the potential for an underestimation of wear volume as the contact area of the implant components change throughout flexion. The purpose of this thesis was to describe, validate, and utilize new advanced imaging techniques to measure TKA implant wear for both in vitro and in vivo applications. Micro-computed tomography (micro-CT), a non-destructive, high resolution imaging technique was utilized to provide detailed images of the geometry of tibial inserts used in wear simulator trials or retrieved from patients, and create surface deviation maps to accurately quantify wear. Ways to create an unworn reference geometry, for use in comparing to a worn retrieved tibial insert when the pre-wear geometry is unknown, were evaluated and a best practice approach was determined. These methods were then applied to study a group of tibial inserts retrieved from patients during revision surgery, which were found to be well functioning with a yearly wear rate equivalent to other contemporary implant designs. Finally, a pilot study to evaluate the use of dynamic single-plane flat panel digital radiography for use in measuring TKA implant wear in vivo was conducted. The system was determined to have a measurement accuracy and precision sufficient to begin a pilot clinical study with patients

ROBUST DEEP LEARNING METHODS FOR SOLVING INVERSE PROBLEMS IN MEDICAL IMAGING

The medical imaging field has a long history of incorporating machine learning algorithms to address inverse problems in image acquisition and analysis. With the impressive successes of deep neural networks on natural images, we seek to answer the obvious question: do these successes also transfer to the medical image domain? The answer may seem straightforward on the surface. Tasks like image-to-image transformation, segmentation, detection, etc., have direct applications for medical images. For example, metal artifact reduction for Computed Tomography (CT) and reconstruction from undersampled k-space signal for Magnetic Resonance (MR) imaging can be formulated as an image-to-image transformation; lesion/tumor detection and segmentation are obvious applications for higher level vision tasks. While these tasks may be similar in formulation, many practical constraints and requirements exist in solving these tasks for medical images. Patient data is highly sensitive and usually only accessible from individual institutions. This creates constraints on the available groundtruth, dataset size, and computational resources in these institutions to train performant models. Due to the mission-critical nature in healthcare applications, requirements such as performance robustness and speed are also stringent. As such, the big-data, dense-computation, supervised learning paradigm in mainstream deep learning is often insufficient to address these situations. In this dissertation, we investigate ways to benefit from the powerful representational capacity of deep neural networks while still satisfying the above-mentioned constraints and requirements. The first part of this dissertation focuses on adapting supervised learning to account for variations such as different medical image modality, image quality, architecture designs, tasks, etc. The second part of this dissertation focuses on improving model robustness on unseen data through domain adaptation, which ameliorates performance degradation due to distribution shifts. The last part of this dissertation focuses on self-supervised learning and learning from synthetic data with a focus in tomographic imaging; this is essential in many situations where the desired groundtruth may not be accessible

Contributions to the improvement of image quality in CBCT and CBμCT and application in the development of a CBμCT system

During the last years cone-beam x-ray CT (CBCT) has been established as a widespread imaging technique and a feasible alternative to conventional CT for dedicated imaging tasks for which the limited flexibility offered by conventional CT advises the development of dedicated designs. CBCT systems are starting to be routinely used in image guided radiotherapy; image guided surgery using C-arms; scan of body parts such as the sinuses, the breast or extremities; and, especially, in preclinical small-animal imaging, often coupled to molecular imaging systems. Despite the research efforts advocated to the advance of CBCT, the challenges introduced by the use of large cone angles and two-dimensional detectors are a field of vigorous research towards the improvement of CBCT image quality. Moreover, systems for small-animal imaging add to the challenges posed by clinical CBCT the need of higher resolution to obtain equivalent image quality in much smaller subjects. This thesis contributes to the progress of CBCT imaging by addressing a variety of issues affecting image quality in CBCT in general and in CBCT for small-animal imaging (CBμCT). As part of this work we have assessed and optimized the performance of CBμCT systems for different imaging tasks. To this end, we have developed a new CBμCT system with variable geometry and all the required software tools for acquisition, calibration and reconstruction. The system served as a tool for the optimization of the imaging process and for the study of image degradation effects in CBμCT, as well as a platform for biological research using small animals. The set of tools for the accurate study of CBCT was completed by developing a fast Monte Carlo simulation engine based on GPUs, specifically devoted to the realistic estimation of scatter and its effects on image quality in arbitrary CBCT configurations, with arbitrary spectra, detector response, and antiscatter grids. This new Monte Carlo engine outperformed current simulation platforms by more than an order of magnitude. Due to the limited options for simulation of spectra in microfocus x-ray sources used in CBμCT, we contributed in this thesis a new spectra generation model based on an empirical model for conventional radiology and mammography sources modified in accordance to experimental data. The new spectral model showed good agreement with experimental exposure and attenuation data for different materials. The developed tools for CBμCT research were used for the study of detector performance in terms of dynamic range. The dynamic range of the detector was characterized together with its effect on image quality. As a result, a new simple method for the extension of the dynamic range of flat-panel detectors was proposed and evaluated. The method is based on a modified acquisition process and a mathematical treatment of the acquired data. Scatter is usually identified as one of the major causes of image quality degradation in CBCT. For this reason the developed Monte Carlo engine was applied to the in-depth study of the effects of scatter for a representative range of CBCT embodiments used in the clinical and preclinical practice. We estimated the amount and spatial distribution of the total scatter fluence and the individual components within. The effect of antiscatter grids in improving image quality and in noise was also evaluated. We found a close relation between scatter and the air gap of the system, in line with previous results in the literature. We also observed a non-negligible contribution of forward-directed scatter that is responsible to a great extent for streak artifacts in CBCT. The spatial distribution of scatter was significantly affected by forward scatter, somewhat challenging the usual assumption that the scatter distribution mostly contains low-frequencies. Antiscatter grids showed to be effective for the reduction of cupping, but they showed a much lower performance when dealing with streaks and a shift toward high frequencies of the scatter distributions. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------A lo largo de los últimos años, el TAC de rayos X de haz cónico (CBCT, de “conebeam” CT) se ha posicionado como una de las técnicas de imagen más ampliamente usadas. El CBCT se ha convertido en una alternativa factible al TAC convencional en tareas de imagen específicas para las que la flexibilidad limitada ofrecida por este hace recomendable el desarrollo de sistemas de imagen dedicados. De esta forma, el CBCT está empezando a usarse de forma rutinaria en varios campos entre los que se incluyen la radioterapia guiada por imagen, la cirugía guiada por imagen usando arcos en C, imagen de partes de la anatomía en las que el TAC convencional no es apropiado, como los senos nasales, las extremidades o la mama, y, especialmente el campo de imagen preclínica con pequeño animal. Los sistemas CBCT usados en este último campo se encuentran habitualmente combinados con sistemas de imagen molecular. A pesar del trabajo de investigación dedicado al avance de la técnica CBCT en los últimos años, los retos introducidos por el uso de haces cónicos y de detectores bidimensionales son un campo candente para la investigación médica, con el objetivo de obtener una calidad de imagen equivalente o superior a la proporcionada por el TAC convencional. En el caso de imagen preclínica, a los retos generados por el uso de CBCT se une la necesidad de una mayor resolución de imagen que permita observar estructuras anatómicas con el mismo nivel de detalle obtenido para humanos. Esta tesis contribuye al progreso del CBCT mediante el estudio de usa serie de efectos que afectan a la calidad de imagen de CBCT en general y en el ámbito preclínico en particular. Como parte de este trabajo, hemos evaluado y optimizado el rendimiento de sistemas CBCT preclínicos en función de la tarea de imagen concreta. Con este fin se ha desarrollado un sistema CBCT para pequeños animales con geometría variable y todas las herramientas necesarias para la adquisición, calibración y reconstrucción de imagen. El sistema sirve como base para la optimización de protocolos de adquisición y para el estudio de fuentes de degradación de imagen además de constituir una plataforma para la investigación biológica en pequeño animal. El conjunto de herramientas para el estudio del CBCT se completó con el desarrollo de una plataforma acelerada de simulación Monte Carlo basada en GPUs, optimizada para la estimación de radiación dispersa en CBCT y sus efectos en la calidad de imagen. La plataforma desarrollada supera el rendimiento de las actuales en más de un orden de magnitud y permite la inclusión de espectros policromáticos de rayos X, de la respuesta realista del detector y de rejillas antiscatter. Debido a las escasas opciones ofrecidas por la literatura para la estimación de espectros de rayos X para fuentes microfoco usadas en imagen preclínica, en esta tesis se incluye el desarrollo de un nuevo modelo de generación de espectros, basado en un modelo existente para fuentes usadas en radiología y mamografía. El modelo fue modificado a partir de datos experimentales. La precisión del modelo presentado se comprobó mediante datos experimentales de exposición y atenuación para varios materiales. Las herramientas desarrolladas se usaron para estudiar el rendimiento de detectores de rayos tipo flat-panel en términos de rango dinámico, explorando los límites impuestos por el mismo en la calidad de imagen. Como resultado se propuso y evaluó un método para la extensión del rango dinámico de este tipo de detectores. El método se basa en la modificación del proceso de adquisición de imagen y en una etapa de postproceso de los datos adquiridos. El simulador Monte Carlo se empleó para el estudio detallado de la naturaleza, distribución espacial y efectos de la radiación dispersa en un rango de sistemas CBCT que cubre el espectro de aplicaciones propuestas en el entorno clínico y preclínico. Durante el estudio se inspeccionó la cantidad y distribución espacial de radiación dispersa y de sus componentes individuales y el efecto causado por la inclusión de rejillas antiscatter en términos de mejora de calidad de imagen y de ruido en la imagen. La distribución de radiación dispersa mostró una acentuada relación con la distancia entre muestra y detector en el equipo, en línea con resultados publicados previamente por otros autores. También se encontró una influencia no despreciable de componentes de radiación dispersa con bajos ángulos de desviación, poniendo en tela de juicio la tradicional asunción que considera que la distribución espacial de la radiación dispersa está formada casi exclusivamente por componentes de muy baja frecuencia. Las rejillas antiscatter demostraron ser efectivas para la reducción del artefacto de cupping, pero su efectividad para tratar artefactos en forma de línea (principalmente formados por radiación dispersa con bajo ángulo de desviación) resultó mucho menor. La inclusión de estas rejillas también enfatiza las componentes de alta frecuencia de la distribución espacial de la radiación dispersa