Augmented Reality Assistance for Surgical Interventions using Optical See-Through Head-Mounted Displays

Augmented Reality (AR) offers an interactive user experience via enhancing the real world environment with computer-generated visual cues and other perceptual information. It has been applied to different applications, e.g. manufacturing, entertainment and healthcare, through different AR media. An Optical See-Through Head-Mounted Display (OST-HMD) is a specialized hardware for AR, where the computer-generated graphics can be overlaid directly onto the user's normal vision via optical combiners. Using OST-HMD for surgical intervention has many potential perceptual advantages. As a novel concept, many technical and clinical challenges exist for OST-HMD-based AR to be clinically useful, which motivates the work presented in this thesis. From the technical aspects, we first investigate the display calibration of OST-HMD, which is an indispensable procedure to create accurate AR overlay. We propose various methods to reduce the user-related error, improve robustness of the calibration, and remodel the calibration as a 3D-3D registration problem. Secondly, we devise methods and develop hardware prototype to increase the user's visual acuity of both real and virtual content through OST-HMD, to aid them in tasks that require high visual acuity, e.g. dental procedures. Thirdly, we investigate the occlusion caused by the OST-HMD hardware, which limits the user's peripheral vision. We propose to use alternative indicators to remind the user of unattended environment motion. From the clinical perspective, we identified many clinical use cases where OST-HMD-based AR is potentially helpful, developed applications integrated with current clinical systems, and conducted proof-of-concept evaluations. We first present a "virtual monitor'' for image-guided surgery. It can replace real radiology monitors in the operating room with easier user control and more flexibility in positioning. We evaluated the "virtual monitor'' for simulated percutaneous spine procedures. Secondly, we developed ARssist, an application for the bedside assistant in robotic surgery. The assistant can see the robotic instruments and endoscope within the patient body with ARssist. We evaluated the efficiency, safety and ergonomics of the assistant during two typical tasks: instrument insertion and manipulation. The performance for inexperienced users is significantly improved with ARssist, and for experienced users, the system significantly enhanced their confidence level. Lastly, we developed ARAMIS, which utilizes real-time 3D reconstruction and visualization to aid the laparoscopic surgeon. It demonstrates the concept of "X-ray see-through'' surgery. Our preliminary evaluation validated the application via a peg transfer task, and also showed significant improvement in hand-eye coordination. Overall, we have demonstrated that OST-HMD based AR application provides ergonomic improvements, e.g. hand-eye coordination. In challenging situations or for novice users, the improvements in ergonomic factors lead to improvement in task performance. With continuous effort as a community, optical see-through augmented reality technology will be a useful interventional aid in the near future

Augmented reality (AR) for surgical robotic and autonomous systems: State of the art, challenges, and solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future

Augmented Reality (AR) for Surgical Robotic and Autonomous Systems: State of the Art, Challenges, and Solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human–robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future

PRELIMINARY FINDINGS OF A POTENZIATED PIEZOSURGERGICAL DEVICE AT THE RABBIT SKULL

The number of available ultrasonic osteotomes has remarkably increased. In vitro and in vivo studies have revealed differences between conventional osteotomes, such as rotating or sawing devices, and ultrasound-supported osteotomes (Piezosurgery®) regarding the micromorphology and roughness values of osteotomized bone surfaces. Objective: the present study compares the micro-morphologies and roughness values of osteotomized bone surfaces after the application of rotating and sawing devices, Piezosurgery Medical® and Piezosurgery Medical New Generation Powerful Handpiece. Methods: Fresh, standard-sized bony samples were taken from a rabbit skull using the following osteotomes: rotating and sawing devices, Piezosurgery Medical® and a Piezosurgery Medical New Generation Powerful Handpiece. The required duration of time for each osteotomy was recorded. Micromorphologies and roughness values to characterize the bone surfaces following the different osteotomy methods were described. The prepared surfaces were examined via light microscopy, environmental surface electron microscopy (ESEM), transmission electron microscopy (TEM), confocal laser scanning microscopy (CLSM) and atomic force microscopy. The selective cutting of mineralized tissues while preserving adjacent soft tissue (dura mater and nervous tissue) was studied. Bone necrosis of the osteotomy sites and the vitality of the osteocytes near the sectional plane were investigated, as well as the proportion of apoptosis or cell degeneration. Results and Conclusions: The potential positive effects on bone healing and reossification associated with different devices were evaluated and the comparative analysis among the different devices used was performed, in order to determine the best osteotomes to be employed during cranio-facial surgery

Body sensor networks: smart monitoring solutions after reconstructive surgery

Advances in reconstructive surgery are providing treatment options in the face of major trauma and cancer. Body Sensor Networks (BSN) have the potential to offer smart solutions to a range of clinical challenges. The aim of this thesis was to review the current state of the art devices, then develop and apply bespoke technologies developed by the Hamlyn Centre BSN engineering team supported by the EPSRC ESPRIT programme to deliver post-operative monitoring options for patients undergoing reconstructive surgery. A wireless optical sensor was developed to provide a continuous monitoring solution for free tissue transplants (free flaps). By recording backscattered light from 2 different source wavelengths, we were able to estimate the oxygenation of the superficial microvasculature. In a custom-made upper limb pressure cuff model, forearm deoxygenation measured by our sensor and gold standard equipment showed strong correlations, with incremental reductions in response to increased cuff inflation durations. Such a device might allow early detection of flap failure, optimising the likelihood of flap salvage. An ear-worn activity recognition sensor was utilised to provide a platform capable of facilitating objective assessment of functional mobility. This work evolved from an initial feasibility study in a knee replacement cohort, to a larger clinical trial designed to establish a novel mobility score in patients recovering from open tibial fractures (OTF). The Hamlyn Mobility Score (HMS) assesses mobility over 3 activities of daily living: walking, stair climbing, and standing from a chair. Sensor-derived parameters including variation in both temporal and force aspects of gait were validated to measure differences in performance in line with fracture severity, which also matched questionnaire-based assessments. Monitoring the OTF cohort over 12 months with the HMS allowed functional recovery to be profiled in great detail. Further, a novel finding of continued improvements in walking quality after a plateau in walking quantity was demonstrated objectively. The methods described in this thesis provide an opportunity to revamp the recovery paradigm through continuous, objective patient monitoring along with self-directed, personalised rehabilitation strategies, which has the potential to improve both the quality and cost-effectiveness of reconstructive surgery services.Open Acces

Yale Medicine : Alumni Bulletin of the School of Medicine, Fall 1990- Summer 1992

This volume contains Yale medicine: alumni bulletin of the School of Medicine, v.25 (Fall 1990) through v.26 (Summer 1992). Prepared in cooperation with the alumni and development offices at the School of Medicine. Earlier volumes are called Yale School of Medicine alumni bulletins, dating from v.1 (1953) through v.13 (1965). Digitized with funding from the Arcadia fund, 2017.https://elischolar.library.yale.edu/yale_med_alumni_newsletters/1010/thumbnail.jp

Natural Sciences in Archaeology and Cultural Heritage

A Special Issue of the international journal Sustainability under the section Sustainability of Culture & Heritage has been made, entitled Natural Sciences in Archaeology and Cultural Heritage. The bridge between science/technology and the humanities (archaeology, anthropology, history of art, and cultural heritage) has formed a well-established interdisciplinary subject with several sub-disciplines; it is growing exponentially, spurred by the fast development of technology in other fields (space exploration, medical, military, and industrial applications). On the other hand, art and culture struggle to survive due to neglect, lack of funding, or the dangers of events such as natural disasters and war. This volume strengthens and exerts the documentation of the sustainability of the issue that arises from the outcome of resulting research and the application of such a duality link. The sustainable dimension emerges from society, education, and economics through the impact of cultural growth, all of which produce a balanced society, in which prosperity, harmony, and development are merged at a sustainable local/regional/national/social level. A wide range of subjects linking the applied natural sciences with archaeology and the cultural heritage of innovative research and applications are presented in this volume

Credulous Spectatorship from Zeuxis to Barthes

This dissertation explores intersections between trompe l\u27oeil painting and photography. It began as an interest in contemporary photographers, such as Thomas Demand, whose photographs of constructed paper models encourage viewers to discover the nature of his interventions. His strategy resonates with a centuries-old strategy in trompe l\u27oeil painting, but now in the terms of photographic, rather than pictorial presence. That is, most of Demand\u27s photographs do not compel the viewer\u27s belief in the tangible presence of the object represented; instead, they exploit photography\u27s indexical promise of delivering the world as it once appeared, in order to temporarily trick viewers about the terms of that indexical delivery. Beyond intersections in artistic strategies, I track reception accounts of trompe l\u27oeil painting and photography for their reliance on a credulous spectator. Pliny\u27s Zeuxis, who is tricked by Parrhasius\u27s painting of a curtain, remains the model for this errant credulity. In their efforts to reveal the manipulation of photographs, historians and theorists assume that the natural attitude for viewing photographs is wholly credulous and recast postmodern viewers as contemporary Zeuxises. Instead of admonishing spectators for such credulity, I argue that trompe l\u27oeil facilitates a pleasurable experience of oscillation between belief and disbelief. I also suggest that these trompe l’oeil deployments of oscillation tend to coincide with historical moments of perceived change in visual technologies—changes due to digitalization, as well as mechanical or other forms of reproduction. Trompe l\u27oeil artists play upon our supposed willingness to accept reproductions for the objects they represent. The inclusion of photographs and/or engravings in these trompe l’oeil paintings simultaneously stages and reprimands our desire for the aura of the actual object. Finally, I suggest that a contemporary renewal of trompe l\u27oeil in the medium of photography reveals an interest in recuperating belief in photographs—a belief not unlike that which Roland Barthes narrates in Camera Lucida. Just as Barthes can discover something of photography\u27s indexical promise, even after decades of his own scholarly efforts to unveil photography\u27s rhetoric of construction, so might we, even while heeding the postmodernist lessons of disbelief, recuperate a moment of belief in a skeptical age

Credulous Spectatorship from Zeuxis to Barthes

This dissertation explores intersections between trompe l\u27oeil painting and photography. It began as an interest in contemporary photographers, such as Thomas Demand, whose photographs of constructed paper models encourage viewers to discover the nature of his interventions. His strategy resonates with a centuries-old strategy in trompe l\u27oeil painting, but now in the terms of photographic, rather than pictorial presence. That is, most of Demand\u27s photographs do not compel the viewer\u27s belief in the tangible presence of the object represented; instead, they exploit photography\u27s indexical promise of delivering the world as it once appeared, in order to temporarily trick viewers about the terms of that indexical delivery. Beyond intersections in artistic strategies, I track reception accounts of trompe l\u27oeil painting and photography for their reliance on a credulous spectator. Pliny\u27s Zeuxis, who is tricked by Parrhasius\u27s painting of a curtain, remains the model for this errant credulity. In their efforts to reveal the manipulation of photographs, historians and theorists assume that the natural attitude for viewing photographs is wholly credulous and recast postmodern viewers as contemporary Zeuxises. Instead of admonishing spectators for such credulity, I argue that trompe l\u27oeil facilitates a pleasurable experience of oscillation between belief and disbelief. I also suggest that these trompe l’oeil deployments of oscillation tend to coincide with historical moments of perceived change in visual technologies—changes due to digitalization, as well as mechanical or other forms of reproduction. Trompe l\u27oeil artists play upon our supposed willingness to accept reproductions for the objects they represent. The inclusion of photographs and/or engravings in these trompe l’oeil paintings simultaneously stages and reprimands our desire for the aura of the actual object. Finally, I suggest that a contemporary renewal of trompe l\u27oeil in the medium of photography reveals an interest in recuperating belief in photographs—a belief not unlike that which Roland Barthes narrates in Camera Lucida. Just as Barthes can discover something of photography\u27s indexical promise, even after decades of his own scholarly efforts to unveil photography\u27s rhetoric of construction, so might we, even while heeding the postmodernist lessons of disbelief, recuperate a moment of belief in a skeptical age