Image overlay surgery based on augmented reality : a systematic review

Acknowledgements We thank the staff of the Medical Library of the University of Aberdeen for their advice and Prof. Jennifer Cleland and Dr Jenny Gregory for discussion and support. This work was funded by the Roland Sutton Academic Trust (0053/R/17) and an Elphinstone PhD Scholarship from the University of Aberdeen.Postprin

Review on Augmented Reality in Oral and Cranio-Maxillofacial Surgery: Toward 'Surgery-Specific' Head-Up Displays

In recent years, there has been an increasing interest towards the augmented reality as applied to the surgical field. We conducted a systematic review of literature classifying the augmented reality applications in oral and cranio-maxillofacial surgery (OCMS) in order to pave the way to future solutions that may ease the adoption of AR guidance in surgical practice. Publications containing the terms 'augmented reality' AND 'maxillofacial surgery', and the terms 'augmented reality' AND 'oral surgery' were searched in the PubMed database. Through the selected studies, we performed a preliminary breakdown according to general aspects, such as surgical subspecialty, year of publication and country of research; then, a more specific breakdown was provided according to technical features of AR-based devices, such as virtual data source, visualization processing mode, tracking mode, registration technique and AR display type. The systematic search identified 30 eligible publications. Most studies (14) were in orthognatic surgery, the minority (2) concerned traumatology, while 6 studies were in oncology and 8 in general OCMS. In 8 of 30 studies the AR systems were based on a head-mounted approach using smart glasses or headsets. In most of these cases (7), a video-see-through mode was implemented, while only 1 study described an optical-see-through mode. In the remaining 22 studies, the AR content was displayed on 2D displays (10), full-parallax 3D displays (6) and projectors (5). In 1 case the AR display type is not specified. AR applications are of increasing interest and adoption in oral and cranio-maxillofacial surgery, however, the quality of the AR experience represents the key requisite for a successful result. Widespread use of AR systems in the operating room may be encouraged by the availability of 'surgery-specific' head-mounted devices that should guarantee the accuracy required for surgical tasks and the optimal ergonomics

Current Accuracy of Augmented Reality Neuronavigation Systems: Systematic Review and Meta-Analysis

BACKGROUND Augmented reality neuronavigation (ARN) systems can overlay three-dimensional anatomy and pathology without the need for a two-dimensional external monitor. Accuracy is crucial for their clinical applicability. We performed a systematic review regarding the reported accuracy of ARN systems and compared them with the accuracy of conventional infrared neuronavigation (CIN). OBJECTIVE Explore the current navigation accuracy of ARN systems and compare them with CIN. METHODS Pubmed and Embase were searched for ARN and CIN systems. For ARN: type of system, method of patient-to-image registration, accuracy method and accuracy of the system was noted. For CIN: navigation accuracy, expressed as target registration error (TRE), was noted. A meta-analysis was performed comparing the TRE of ARN and CIN systems. RESULTS 35 studies were included, 12 for ARN and 23 for CIN. ARN systems were divided into head-mounted display and heads-up display. In ARN, four methods were encountered for patient-to-image registration, of which point-pair matching was the one most frequently used. Five methods for assessing accuracy were described. 94 TRE measurements of ARN systems were compared with 9058 TRE measurements of CIN systems. Mean TRE was 2.5 mm (CI 95% 0.7 - 4.4) for ARN systems and 2.6 mm (CI 95% 2.1 - 3.1) for CIN systems. CONCLUSIONS In ARN, there seems to be lack of agreement regarding the best method to assess accuracy. Nevertheless, ARN systems seem able to achieve an accuracy comparable with CIN systems. Future studies should be prospective and compare TREs which should be measured in a standardized fashion

Design and clinical evaluation of an image-guided surgical microscope with an integrated tracking system

A new image-guided microscope system using augmented reality image overlays has been developed. With this system, CT cut-views and segmented objects such as tumors that have been previously extracted from preoperative tomographic images can be directly displayed as augmented reality overlays on the microscope image. The novelty of this design stems from the inclusion of a precise mini-tracker directly on the microscope. This device, which is rigidly mounted to the microscope, is used to track the movements of surgical tools and the patient. In addition to an accuracy gain, this setup offers improved ergonomics since it is much easier for the surgeon to keep an unobstructed line of sight to tracked objects. We describe the components of the system: microscope calibration, image registration, tracker assembly and registration, tool tracking, and augmented reality display. The accuracy of the system has been measured by validation on plastic skulls and cadaver heads, obtaining an overlay error of 0.7mm. In addition, a numerical simulation of the system has been done in order to complement the accuracy study, showing that the integration of the tracker onto the microscope could lead to an improvement of the accuracy to the order of 0.5mm. Finally, we describe our clinical experience using the system in the operation room, where three operations have been performed to dat

Design and evaluation of an augmented reality simulator using leap motion

Advances in virtual and augmented reality (AR) are having an impact on the medical field in areas such as surgical simulation. Improvements to surgical simulation will provide students and residents with additional training and evaluation methods. This is particularly important for procedures such as the endoscopic third ventriculostomy (ETV), which residents perform regularly. Simulators such as NeuroTouch, have been designed to aid in training associated with this procedure. The authors have designed an affordable and easily accessible ETV simulator, and compare it with the existing NeuroTouch for its usability and training effectiveness. This simulator was developed using Unity, Vuforia and the leap motion (LM) for an AR environment. The participants, 16 novices and two expert neurosurgeons, were asked to complete 40 targeting tasks. Participants used the NeuroTouch tool or a virtual hand controlled by the LM to select the position and orientation for these tasks. The length of time to complete each task was recorded and the trajectory log files were used to calculate performance. The resulting data from the novices\u27 and experts\u27 speed and accuracy are compared, and they discuss the objective performance of training in terms of the speed and accuracy of targeting accuracy for each system

Intra-operative applications of augmented reality in glioma surgery: a systematic review

BackgroundAugmented reality (AR) is increasingly being explored in neurosurgical practice. By visualizing patient-specific, three-dimensional (3D) models in real time, surgeons can improve their spatial understanding of complex anatomy and pathology, thereby optimizing intra-operative navigation, localization, and resection. Here, we aimed to capture applications of AR in glioma surgery, their current status and future potential.MethodsA systematic review of the literature was conducted. This adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline. PubMed, Embase, and Scopus electronic databases were queried from inception to October 10, 2022. Leveraging the Population, Intervention, Comparison, Outcomes, and Study design (PICOS) framework, study eligibility was evaluated in the qualitative synthesis. Data regarding AR workflow, surgical application, and associated outcomes were then extracted. The quality of evidence was additionally examined, using hierarchical classes of evidence in neurosurgery.ResultsThe search returned 77 articles. Forty were subject to title and abstract screening, while 25 proceeded to full text screening. Of these, 22 articles met eligibility criteria and were included in the final review. During abstraction, studies were classified as “development” or “intervention” based on primary aims. Overall, AR was qualitatively advantageous, due to enhanced visualization of gliomas and critical structures, frequently aiding in maximal safe resection. Non-rigid applications were also useful in disclosing and compensating for intra-operative brain shift. Irrespective, there was high variance in registration methods and measurements, which considerably impacted projection accuracy. Most studies were of low-level evidence, yielding heterogeneous results.ConclusionsAR has increasing potential for glioma surgery, with capacity to positively influence the onco-functional balance. However, technical and design limitations are readily apparent. The field must consider the importance of consistency and replicability, as well as the level of evidence, to effectively converge on standard approaches that maximize patient benefit

Roadmap on 3D integral imaging: Sensing, processing, and display

This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field

Computer Mediated Reality Technologies: A Conceptual Framework and Survey of the State of the Art in Healthcare Intervention

The trend of an ageing and growing world population, particularly in developed countries, is expected to continue for decades to come causing an increase in demand for healthcare resources and services. Consequently, demand is growing faster than rises in funding. The UK government, in partnership with the European Commission’s Vision for 2020, propose a paradigm shift towards the delivery of more patient-centred self-care interventions, facilitated by novel ubiquitous computer mediated reality technology applications, as a key strategy to overcome the scarcity of health resources gap. If this vision is to become a reality, it is crucial that state of the art research focuses efforts on the development of applications that support the delivery of patient-centred self-care interventions

Markerless Augmented Reality via Stereo Video See-Through Head-Mounted Display Device

Conventionally, the camera localization for augmented reality (AR) relies on detecting a known pattern within the captured images. In this study, a markerless AR scheme has been designed based on a Stereo Video See-Through Head-Mounted Display (HMD) device. The proposed markerless AR scheme can be utilized for medical applications such as training, telementoring, or preoperative explanation. Firstly, a virtual model for AR visualization is aligned to the target in physical space by an improved Iterative Closest Point (ICP) based surface registration algorithm, with the target surface structure reconstructed by a stereo camera pair; then, a markerless AR camera localization method is designed based on the Kanade-Lucas-Tomasi (KLT) feature tracking algorithm and the Random Sample Consensus (RANSAC) correction algorithm. Our AR camera localization method is shown to be better than the traditional marker-based and sensor-based AR environment. The demonstration system was evaluated with a plastic dummy head and the display result is satisfactory for a multiple-view observation

Exploration and Implementation of Augmented Reality for External Beam Radiotherapy

We have explored applications of Augmented Reality (AR) for external beam radiotherapy to assist with treatment planning, patient education, and treatment delivery. We created an AR development framework for applications in radiotherapy (RADiotherapy Augmented Reality, RAD-AR) for AR ready consumer electronics such as tablet computers and head mounted devices (HMD). We implemented in RAD-AR three tools to assist radiotherapy practitioners with: treatment plans evaluation, patient pre-treatment information/education, and treatment delivery. We estimated accuracy and precision of the patient setup tool and the underlying self-tracking technology, and fidelity of AR content geometric representation, on the Apple iPad tablet computer and the Microsoft HoloLens HMD. Results showed that the technology could already be applied for detection of large treatment setup errors, and could become applicable to other aspects of treatment delivery subject to technological improvements that can be expected in the near future. We performed user feedback studies of the patient education and the plan evaluation tools. Results indicated an overall positive user evaluation of AR technology compared to conventional tools for the radiotherapy elements implemented. We conclude that AR will become a useful tool in radiotherapy bringing real benefits for both clinicians and patients, contributing to successful treatment outcomes