Utilizing image guided surgery for user interaction in medical augmented reality

The graphical overlay of additional medical information over the patient during a surgical procedure has long been considered one of the most promising applications of augmented reality. While many experimental systems for augmented reality in medicine have reached an advanced state and can deliver high-quality augmented video streams, they usually depend heavily on specialized dedicated hardware. Such dedicated system components, which originally have been designed for engineering applications or VR research, often are ill-suited for use in the clinical practice. We have described a novel medical augmented reality application, which is based almost exclusively on existing, commercially available, and certified medical equipment. In our system, a so-called image guided surgery device is used for tracking a webcam, which delivers the digital video stream of the physical scene that is augmented with the virtual information. In this paper, we show how the capability of the image guided surgery system for tracking surgical instruments can be harnessed for user interaction. Our method enables the user to define points and freely drawn shapes in 3-d and provides selectable menu items, which can be located in immediate proximity to the patient. This eliminates the need for conventional touchscreen- or mouse-based user interaction without requiring additional dedicated hardware like dedicated tracking systems or specialized 3-d input devices. Thus the surgeon can directly interact with the system, without the help of additional personnel. We demonstrate our new input method with an application for creating operation plan sketches directly on the patient in an augmented view

Augmented Reality Assistance for Surgical Interventions using Optical See-Through Head-Mounted Displays

Augmented Reality (AR) offers an interactive user experience via enhancing the real world environment with computer-generated visual cues and other perceptual information. It has been applied to different applications, e.g. manufacturing, entertainment and healthcare, through different AR media. An Optical See-Through Head-Mounted Display (OST-HMD) is a specialized hardware for AR, where the computer-generated graphics can be overlaid directly onto the user's normal vision via optical combiners. Using OST-HMD for surgical intervention has many potential perceptual advantages. As a novel concept, many technical and clinical challenges exist for OST-HMD-based AR to be clinically useful, which motivates the work presented in this thesis. From the technical aspects, we first investigate the display calibration of OST-HMD, which is an indispensable procedure to create accurate AR overlay. We propose various methods to reduce the user-related error, improve robustness of the calibration, and remodel the calibration as a 3D-3D registration problem. Secondly, we devise methods and develop hardware prototype to increase the user's visual acuity of both real and virtual content through OST-HMD, to aid them in tasks that require high visual acuity, e.g. dental procedures. Thirdly, we investigate the occlusion caused by the OST-HMD hardware, which limits the user's peripheral vision. We propose to use alternative indicators to remind the user of unattended environment motion. From the clinical perspective, we identified many clinical use cases where OST-HMD-based AR is potentially helpful, developed applications integrated with current clinical systems, and conducted proof-of-concept evaluations. We first present a "virtual monitor'' for image-guided surgery. It can replace real radiology monitors in the operating room with easier user control and more flexibility in positioning. We evaluated the "virtual monitor'' for simulated percutaneous spine procedures. Secondly, we developed ARssist, an application for the bedside assistant in robotic surgery. The assistant can see the robotic instruments and endoscope within the patient body with ARssist. We evaluated the efficiency, safety and ergonomics of the assistant during two typical tasks: instrument insertion and manipulation. The performance for inexperienced users is significantly improved with ARssist, and for experienced users, the system significantly enhanced their confidence level. Lastly, we developed ARAMIS, which utilizes real-time 3D reconstruction and visualization to aid the laparoscopic surgeon. It demonstrates the concept of "X-ray see-through'' surgery. Our preliminary evaluation validated the application via a peg transfer task, and also showed significant improvement in hand-eye coordination. Overall, we have demonstrated that OST-HMD based AR application provides ergonomic improvements, e.g. hand-eye coordination. In challenging situations or for novice users, the improvements in ergonomic factors lead to improvement in task performance. With continuous effort as a community, optical see-through augmented reality technology will be a useful interventional aid in the near future

Mixed reality simulation of rasping procedure in artificial cervical disc replacement (ACDR) surgery

<p>Abstract</p> <p>Background</p> <p>Until quite recently spinal disorder problems in the U.S. have been operated by fusing cervical vertebrae instead of replacement of the cervical disc with an artificial disc. Cervical disc replacement is a recently approved procedure in the U.S. It is one of the most challenging surgical procedures in the medical field due to the deficiencies in available diagnostic tools and insufficient number of surgical practices For physicians and surgical instrument developers, it is critical to understand how to successfully deploy the new artificial disc replacement systems. Without proper understanding and practice of the deployment procedure, it is possible to injure the vertebral body. Mixed reality (MR) and virtual reality (VR) surgical simulators are becoming an indispensable part of physicians’ training, since they offer a risk free training environment. In this study, MR simulation framework and intricacies involved in the development of a MR simulator for the rasping procedure in artificial cervical disc replacement (ACDR) surgery are investigated. The major components that make up the MR surgical simulator with motion tracking system are addressed. </p> <p>Findings</p> <p>A mixed reality surgical simulator that targets rasping procedure in the artificial cervical disc replacement surgery with a VICON motion tracking system was developed. There were several challenges in the development of MR surgical simulator. First, the assembly of different hardware components for surgical simulation development that involves knowledge and application of interdisciplinary fields such as signal processing, computer vision and graphics, along with the design and placements of sensors etc . Second challenge was the creation of a physically correct model of the rasping procedure in order to attain critical forces. This challenge was handled with finite element modeling. The third challenge was minimization of error in mapping movements of an actor in real model to a virtual model in a process called registration. This issue was overcome by a two-way (virtual object to real domain and real domain to virtual object) semi-automatic registration method.</p> <p>Conclusions</p> <p>The applicability of the VICON MR setting for the ACDR surgical simulator is demonstrated. The main stream problems encountered in MR surgical simulator development are addressed. First, an effective environment for MR surgical development is constructed. Second, the strain and the stress intensities and critical forces are simulated under the various rasp instrument loadings with impacts that are applied on intervertebral surfaces of the anterior vertebrae throughout the rasping procedure. Third, two approaches are introduced to solve the registration problem in MR setting. Results show that our system creates an effective environment for surgical simulation development and solves tedious and time-consuming registration problems caused by misalignments. Further, the MR ACDR surgery simulator was tested by 5 different physicians who found that the MR simulator is effective enough to teach the anatomical details of cervical discs and to grasp the basics of the ACDR surgery and rasping procedure</p

Optimised Calibration, Registration and Tracking for Image Enhanced Surgical Navigation in ENT Operations

EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Entornos multimedia de realidad aumentada en el campo del arte

La relación ente Ciencia y Arte ha mantenido a lo largo de la historia momentos de proximidad o distanciamiento, llegando a entenderse como dos culturas diferentes, pero también se han producido situaciones interdisciplinares de colaboración e intercambio que en nuestros días mantienen como nexo común la cultura digital y el uso del ordenador. Según Berenguer (2002) desde la aparición del ordenador, científicos y artistas están encontrando un espacio común de trabajo y entendimiento. Mediante el empleo de las nuevas tecnologías, la distancia que separa ambas disciplinas es cada vez más corta. En esta tesis, cuyo título es "Entornos Multimedia de Realidad Aumentada en el Campo del Arte", se presenta una investigación teórico-práctica de la tecnología de realidad aumentada aplicada al arte y campos afines, como el edutainment (educación + entretenimiento). La investigación se ha realizado en dos bloques: en el primer bloque se trata la tecnología desde distintos factores que se han considerado relevantes para su entendimiento y funcionamiento; en el segundo se presentan un total de seis ensayos que constituyen la parte práctica de esta tesis.Portalés Ricart, C. (2008). Entornos multimedia de realidad aumentada en el campo del arte [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/3402Palanci

Optimised calibration, registration and tracking for image enhanced surgical navigation in ENT operations

EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Mobile and Low-cost Hardware Integration in Neurosurgical Image-Guidance

It is estimated that 13.8 million patients per year require neurosurgical interventions worldwide, be it for a cerebrovascular disease, stroke, tumour resection, or epilepsy treatment, among others. These procedures involve navigating through and around complex anatomy in an organ where damage to eloquent healthy tissue must be minimized. Neurosurgery thus has very specific constraints compared to most other domains of surgical care. These constraints have made neurosurgery particularly suitable for integrating new technologies. Any new method that has the potential to improve surgical outcomes is worth pursuing, as it has the potential to not only save and prolong lives of patients, but also increase the quality of life post-treatment. In this thesis, novel neurosurgical image-guidance methods are developed, making use of currently available, low-cost off-the-shelf components. In particular, a mobile device (e.g. smartphone or tablet) is integrated into a neuronavigation framework to explore new augmented reality visualization paradigms and novel intuitive interaction methods. The developed tools aim at improving image-guidance using augmented reality to improve intuitiveness and ease of use. Further, we use gestures on the mobile device to increase interactivity with the neuronavigation system in order to provide solutions to the problem of accuracy loss or brain shift that occurs during surgery. Lastly, we explore the effectiveness and accuracy of low-cost hardware components (i.e. tracking systems and ultrasound) that could be used to replace the current high cost hardware that are integrated into commercial image-guided neurosurgery systems. The results of our work show the feasibility of using mobile devices to improve neurosurgical processes. Augmented reality enables surgeons to focus on the surgical field while getting intuitive guidance information. Mobile devices also allow for easy interaction with the neuronavigation system thus enabling surgeons to directly interact with systems in the operating room to improve accuracy and streamline procedures. Lastly, our results show that low-cost components can be integrated into a neurosurgical guidance system at a fraction of the cost, while having a negligible impact on accuracy. The developed methods have the potential to improve surgical workflows, as well as democratize access to higher quality care worldwide

Wissensbasierte Situationsinterpretation für eine kontextbezogene Chirurgieassistenz mittels Erweiterter Realität

Chirurgische Assistenzsysteme unterstützen den Chirurgen bei der Durchführung einer Operation. Durch ihren Einsatz sollen das Patientenrisiko gesenkt und die chirurgische Belastung verringert werden. Ein Ziel zukünftiger Assistenzsysteme ist die Bereitstellung einer kontextbezogenen Unterstützung des Chirurgen. Im Fokus der vorliegenden Arbeit stehen ein Konzept und die zugehörigen Methoden zur Realisierung solch einer Assistenz

Realistic rendering and reconstruction of astronomical objects and an augmented reality application for astronomy

These days, there is an ever increasing need for realistic models, renderings and visualization of astronomical objects to be used in planetarium and as a tool in modern astrophysical research. One of the major goals of this dissertation is to develop novel algorithms for recovering and rendering 3D models of a specific set of astronomical objects. We first present a method to render the color and shape of the solar disc in different climate conditions as well as for different height to temperature atmospheric profiles. We then present a method to render and reconstruct the 3D distribution of reflection nebulae. The rendering model takes into account scattering and absorption to generate physically realistic visualization of reflection nebulae. Further, we propose a reconstruction method for another type of astronomical objects, planetary nebulae. We also present a novel augmented reality application called the augmented astronomical telescope, tailored for educational astronomy. The real-time application augments the view through a telescope by projecting additional information such as images, text and video related to the currently observed object during observation. All methods previously proposed for rendering and reconstructing astronomical objects can be used to create novel content for the presented augmented reality application.Realistische Modelle, Visualisierungen und Renderings von astronomischen Objekten gewinnen heuzutage in Planetarium Shows oder als Werkzeug für die Astrophysikalische Forschung immer mehr an Bedeutung. Eines der Hauptziele dieser Dissertation ist es, neue Algorithmen zum Rendering und zur Rekonstruktion von Astronomischen Objekten zu entwickeln. Wir beschreiben zuerst ein Verfahren zum Rendering von Farbe und Form der Sonnenscheibe für verschiedene Klimate und gegebenen Höhe zu Temperatur Profilen. Im weiterem wird eine Methode zum Rendering und zur Rekonstruktion von 3D Modellen von Reflexionsnebeln präsentiert. Das Renderingmodell berücksichtigt Streuung und Absorption, um physikalisch realistische Visualisierungen von Reflexionsnebeln zu erzeugen. Weiter, wird ein Rekonstruktionsalgorithmus für eine andere Art astronomischer Objekte, Planetarische Nebel, vorgeschlagen. Wir stellen eine neuartige Erweiterte Realität Anwendung vor, welche für die astronomische Bildung zugeschnitten ist. Die Anwedung erweitert die Sicht durch das Okular des Teleskopes und projiziert zusätzliche Informationen wie Bilder, Text und Video online, während des Betrachtens. Alle vorher erwähnten Verfahren zum Rendering und zur Rekonstruktion von Astronomischen Objekten können verwendet werden, um Inhalte für die vorgestellte Erweiterte Realität Anwendung zu entwerfen

Realistic rendering and reconstruction of astronomical objects and an augmented reality application for astronomy

These days, there is an ever increasing need for realistic models, renderings and visualization of astronomical objects to be used in planetarium and as a tool in modern astrophysical research. One of the major goals of this dissertation is to develop novel algorithms for recovering and rendering 3D models of a specific set of astronomical objects. We first present a method to render the color and shape of the solar disc in different climate conditions as well as for different height to temperature atmospheric profiles. We then present a method to render and reconstruct the 3D distribution of reflection nebulae. The rendering model takes into account scattering and absorption to generate physically realistic visualization of reflection nebulae. Further, we propose a reconstruction method for another type of astronomical objects, planetary nebulae. We also present a novel augmented reality application called the augmented astronomical telescope, tailored for educational astronomy. The real-time application augments the view through a telescope by projecting additional information such as images, text and video related to the currently observed object during observation. All methods previously proposed for rendering and reconstructing astronomical objects can be used to create novel content for the presented augmented reality application.Realistische Modelle, Visualisierungen und Renderings von astronomischen Objekten gewinnen heuzutage in Planetarium Shows oder als Werkzeug für die Astrophysikalische Forschung immer mehr an Bedeutung. Eines der Hauptziele dieser Dissertation ist es, neue Algorithmen zum Rendering und zur Rekonstruktion von Astronomischen Objekten zu entwickeln. Wir beschreiben zuerst ein Verfahren zum Rendering von Farbe und Form der Sonnenscheibe für verschiedene Klimate und gegebenen Höhe zu Temperatur Profilen. Im weiterem wird eine Methode zum Rendering und zur Rekonstruktion von 3D Modellen von Reflexionsnebeln präsentiert. Das Renderingmodell berücksichtigt Streuung und Absorption, um physikalisch realistische Visualisierungen von Reflexionsnebeln zu erzeugen. Weiter, wird ein Rekonstruktionsalgorithmus für eine andere Art astronomischer Objekte, Planetarische Nebel, vorgeschlagen. Wir stellen eine neuartige Erweiterte Realität Anwendung vor, welche für die astronomische Bildung zugeschnitten ist. Die Anwedung erweitert die Sicht durch das Okular des Teleskopes und projiziert zusätzliche Informationen wie Bilder, Text und Video online, während des Betrachtens. Alle vorher erwähnten Verfahren zum Rendering und zur Rekonstruktion von Astronomischen Objekten können verwendet werden, um Inhalte für die vorgestellte Erweiterte Realität Anwendung zu entwerfen