Augmented Reality-based Feedback for Technician-in-the-loop C-arm Repositioning

Interventional C-arm imaging is crucial to percutaneous orthopedic procedures as it enables the surgeon to monitor the progress of surgery on the anatomy level. Minimally invasive interventions require repeated acquisition of X-ray images from different anatomical views to verify tool placement. Achieving and reproducing these views often comes at the cost of increased surgical time and radiation dose to both patient and staff. This work proposes a marker-free "technician-in-the-loop" Augmented Reality (AR) solution for C-arm repositioning. The X-ray technician operating the C-arm interventionally is equipped with a head-mounted display capable of recording desired C-arm poses in 3D via an integrated infrared sensor. For C-arm repositioning to a particular target view, the recorded C-arm pose is restored as a virtual object and visualized in an AR environment, serving as a perceptual reference for the technician. We conduct experiments in a setting simulating orthopedic trauma surgery. Our proof-of-principle findings indicate that the proposed system can decrease the 2.76 X-ray images required per desired view down to zero, suggesting substantial reductions of radiation dose during C-arm repositioning. The proposed AR solution is a first step towards facilitating communication between the surgeon and the surgical staff, improving the quality of surgical image acquisition, and enabling context-aware guidance for surgery rooms of the future. The concept of technician-in-the-loop design will become relevant to various interventions considering the expected advancements of sensing and wearable computing in the near future

A tracker alignment framework for augmented reality

To achieve accurate registration, the transformations which locate the tracking system components with respect to the environment must be known. These transformations relate the base of the tracking system to the virtual world and the tracking system's sensor to the graphics display. In this paper we present a unified, general calibration method for calculating these transformations. A user is asked to align the display with objects in the real world. Using this method, the sensor to display and tracker base to world transformations can be determined with as few as three measurements

Optical versus video see-through mead-mounted displays in medical visualization

We compare two technological approaches to augmented reality for 3-D medical visualization: optical and video see-through devices. We provide a context to discuss the technology by reviewing several medical applications of augmented-reality research efforts driven by real needs in the medical field, both in the United States and in Europe. We then discuss the issues for each approach, optical versus video, from both a technology and human-factor point of view. Finally, we point to potentially promising future developments of such devices including eye tracking and multifocus planes capabilities, as well as hybrid optical/video technology

Virtual reality

Virtual reality is still in its infancy. However, many contemporary applications already have proven virtual reality to be indispensable to everyday life. For instance, the technology of virtual product design and manufacturing makes the new products better and cheaper. The applications of VR in medicine allow doctors to diagnose a disease more accurately. Without a doubt, it has and will foster more innovative research and applications. High-resolution, low-lag and low-price systems will be the focus of future virtual reality research. The technology has been infiltrated into many application fields, involving training, medical, engineering; space exploring and communication.published_or_final_versio

Towards markerless orthopaedic navigation with intuitive Optical See-through Head-mounted displays

The potential of image-guided orthopaedic navigation to improve surgical outcomes has been well-recognised during the last two decades. According to the tracked pose of target bone, the anatomical information and preoperative plans are updated and displayed to surgeons, so that they can follow the guidance to reach the goal with higher accuracy, efficiency and reproducibility. Despite their success, current orthopaedic navigation systems have two main limitations: for target tracking, artificial markers have to be drilled into the bone and calibrated manually to the bone, which introduces the risk of additional harm to patients and increases operating complexity; for guidance visualisation, surgeons have to shift their attention from the patient to an external 2D monitor, which is disruptive and can be mentally stressful. Motivated by these limitations, this thesis explores the development of an intuitive, compact and reliable navigation system for orthopaedic surgery. To this end, conventional marker-based tracking is replaced by a novel markerless tracking algorithm, and the 2D display is replaced by a 3D holographic Optical see-through (OST) Head-mounted display (HMD) precisely calibrated to a user's perspective. Our markerless tracking, facilitated by a commercial RGBD camera, is achieved through deep learning-based bone segmentation followed by real-time pose registration. For robust segmentation, a new network is designed and efficiently augmented by a synthetic dataset. Our segmentation network outperforms the state-of-the-art regarding occlusion-robustness, device-agnostic behaviour, and target generalisability. For reliable pose registration, a novel Bounded Iterative Closest Point (BICP) workflow is proposed. The improved markerless tracking can achieve a clinically acceptable error of 0.95 deg and 2.17 mm according to a phantom test. OST displays allow ubiquitous enrichment of perceived real world with contextually blended virtual aids through semi-transparent glasses. They have been recognised as a suitable visual tool for surgical assistance, since they do not hinder the surgeon's natural eyesight and require no attention shift or perspective conversion. The OST calibration is crucial to ensure locational-coherent surgical guidance. Current calibration methods are either human error-prone or hardly applicable to commercial devices. To this end, we propose an offline camera-based calibration method that is highly accurate yet easy to implement in commercial products, and an online alignment-based refinement that is user-centric and robust against user error. The proposed methods are proven to be superior to other similar State-of- the-art (SOTA)s regarding calibration convenience and display accuracy. Motivated by the ambition to develop the world's first markerless OST navigation system, we integrated the developed markerless tracking and calibration scheme into a complete navigation workflow designed for femur drilling tasks during knee replacement surgery. We verify the usability of our designed OST system with an experienced orthopaedic surgeon by a cadaver study. Our test validates the potential of the proposed markerless navigation system for surgical assistance, although further improvement is required for clinical acceptance.Open Acces

Augmented reality and scene examination

The research presented in this thesis explores the impact of Augmented Reality on human performance, and compares this technology with Virtual Reality using a head-mounted video-feed for a variety of tasks that relate to scene examination. The motivation for the work was the question of whether Augmented Reality could provide a vehicle for training in crime scene investigation. The Augmented Reality application was developed using fiducial markers in the Windows Presentation Foundation, running on a wearable computer platform; Virtual Reality was developed using the Crytek game engine to present a photo-realistic 3D environment; and a video-feed was provided through head-mounted webcam. All media were presented through head-mounted displays of similar resolution to provide the sole source of visual information to participants in the experiments. The experiments were designed to increase the amount of mobility required to conduct the search task, i.e., from rotation in the horizontal or vertical plane through to movement around a room. In each experiment, participants were required to find objects and subsequently recall their location. It is concluded that human performance is affected not merely via the medium through which the world is perceived but moreover, the constraints governing how movement in the world is controlled

The Effectiveness of Monitor-Based Augmented Reality Paradigms for Learning Space-Related Technical Tasks

Currently today there are many types of media that can help individuals learn and excel in the on going effort to acquire knowledge for a specific trait or function in a workplace, laboratory, or learning facility. Technology has advanced in the fields of transportation, information gathering, and education. The need for better recall of information is in demand in a wide variety of areas. Augmented reality (AR) is a technology that may help meet this demand. AR is a hybrid of reality and virtual reality (VR) that uses the three-dimensional location viewed through a video or optical see-through media to capture the object\u27s coordinates and add virtual images, objects, or text superimposed on the scene (Azuma, 1997). The purpose of this research is to investigate four different modes of presentation and the effect of those modes on learning and recall of information using monitor-based Augmented Reality. The four modes of presentation are Select, Observe, Interact and Print modes. Each mode possesses different attributes that may affect learning and recall. The Select mode can be described as a mode of presentation that allows movement of the work piece in front of the tracking camera. The Observe mode involves information presentation using a pre-recorded video scene presented with no interaction with the work piece. The Interact mode allows the user to view a pre-recorded video scene that allows the user to point and click on the component of the work piece with a computer mouse on the monitor. The Print mode consists of printed material of each work piece component. It was hypothesized that the Select mode would provide the user with the richest presentation of information due to information access capabilities helping to decrease work time, reduce the amount of error likelihood during usage, enhance the user\u27s motivation for learning tasks, and increase concurrent learning and performances due to recall and retention. It was predicted that the Select mode would result in trainees that would recall the greatest amount of information even after extended periods of time had elapsed. This hypothesis was not supported. No significant differences between the four groups were found