Ultrasound-Augmented Laparoscopy

Laparoscopic surgery is perhaps the most common minimally invasive procedure for many diseases in the abdomen. Since the laparoscopic camera provides only the surface view of the internal organs, in many procedures, surgeons use laparoscopic ultrasound (LUS) to visualize deep-seated surgical targets. Conventionally, the 2D LUS image is visualized in a display spatially separate from that displays the laparoscopic video. Therefore, reasoning about the geometry of hidden targets requires mentally solving the spatial alignment, and resolving the modality differences, which is cognitively very challenging. Moreover, the mental representation of hidden targets in space acquired through such cognitive medication may be error prone, and cause incorrect actions to be performed. To remedy this, advanced visualization strategies are required where the US information is visualized in the context of the laparoscopic video. To this end, efficient computational methods are required to accurately align the US image coordinate system with that centred in the camera, and to render the registered image information in the context of the camera such that surgeons perceive the geometry of hidden targets accurately. In this thesis, such a visualization pipeline is described. A novel method to register US images with a camera centric coordinate system is detailed with an experimental investigation into its accuracy bounds. An improved method to blend US information with the surface view is also presented with an experimental investigation into the accuracy of perception of the target locations in space

Smart Localization Using a New Sensor Association Framework for Outdoor Augmented Reality Systems

Augmented Reality (AR) aims at enhancing our the real world, by adding fictitious elements that are not perceptible naturally such as: computer-generated images, virtual objects, texts, symbols, graphics, sounds, and smells. The quality of the real/virtual registration depends mainly on the accuracy of the 3D camera pose estimation. In this paper, we present an original real-time localization system for outdoor AR which combines three heterogeneous sensors: a camera, a GPS, and an inertial sensor. The proposed system is subdivided into two modules: the main module is vision based; it estimates the user’s location using a markerless tracking method. When the visual tracking fails, the system switches automatically to the secondary localization module composed of the GPS and the inertial sensor

Augmented Reality in Minimally Invasive Surgery

In the last 15 years Minimally Invasive Surgery, with techniques such as laparoscopy or endoscopy, has become very important and research in this field is increasing since these techniques provide the surgeons with less invasive means of reaching the patient’s internal anatomy and allow for entire procedures to be performed with only minimal trauma to the patient. The advantages of the use of this surgical method are evident for patients because the possible trauma is reduced, postoperative recovery is generally faster and there is less scarring. Despite the improvement in outcomes, indirect access to the operation area causes restricted vision, difficulty in hand-eye coordination, limited mobility handling instruments, two-dimensional imagery with a lack of detailed information and a limited visual field during the whole operation. The use of the emerging Augmented Reality technology shows the way forward by bringing the advantages of direct visualization (which you have in open surgery) back to minimally invasive surgery and increasing the physician's view of his surroundings with information gathered from patient medical images. Augmented Reality can avoid some drawbacks of Minimally Invasive Surgery and can provide opportunities for new medical treatments. After two decades of research into medical Augmented Reality, this technology is now advanced enough to meet the basic requirements for a large number of medical applications and it is feasible that medical AR applications will be accepted by physicians in order to evaluate their use and integration into the clinical workflow. Before seeing the systematic use of these technologies as support for minimally invasive surgery some improvements are still necessary in order to fully satisfy the requirements of operating physicians

A feature-based approach for monocular camera tracking in unknown environments

© 2017 IEEE. Camera tracking is an important issue in many computer vision and robotics applications, such as, augmented reality and Simultaneous Localization And Mapping (SLAM). In this paper, a feature-based technique for monocular camera tracking is proposed. The proposed approach is based on tracking a set of sparse features, which are successively tracked in a stream of video frames. In the developed system, camera initially views a chessboard with known cell size for few frames to be enabled to construct initial map of the environment. Thereafter, Camera pose estimation for each new incoming frame is carried out in a framework that is merely working with a set of visible natural landmarks. Estimation of 6-DOF camera pose parameters is performed using a particle filter. Moreover, recovering depth of newly detected landmarks, a linear triangulation method is used. The proposed method is applied on real world videos and positioning error of the camera pose is less than 3 cm in average that indicates effectiveness and accuracy of the proposed method

A comparative study of the sense of presence and anxiety in an invisible marker versus a marker Augmented Reality system for the treatment of phobia towards small animals

Phobia towards small animals has been treated using exposure in vivo and virtual reality. Recently, augmented reality (AR) has also been presented as a suitable tool. The first AR system developed for this purpose used visible markers for tracking. In this first system, the presence of visible markers warns the user of the appearance of animals. To avoid this warning, this paper presents a second version in which the markers are invisible. First, the technical characteristics of a prototype are described. Second, a comparative study of the sense of presence and anxiety in a non-phobic population using the visible marker-tracking system and the invisible marker-tracking system is presented. Twenty-four participants used the two systems. The participants were asked to rate their anxiety level (from 0 to 10) at 8 different moments. Immediately after their experience, the participants were given the SUS questionnaire to assess their subjective sense of presence. The results indicate that the invisible marker-tracking system induces a similar or higher sense of presence than the visible marker-tracking system, and it also provokes a similar or higher level of anxiety in important steps for therapy. Moreover, 83.33% of the participants reported that they did not have the same sensations/surprise using the two systems, and they scored the advantage of using the invisible marker-tracking system (IMARS) at 5.19 +/- 2.25 (on a scale from 1 to 10). However, if only the group with higher fear levels is considered, 100% of the participants reported that they did not have the same sensations/surprise with the two systems, scoring the advantage of using IMARS at 6.38 +/- 1.60 (on a scale from 1 to 10). (C) 2011 Elsevier Ltd. All rights reserved.Juan, M.; Joele, D. (2011). A comparative study of the sense of presence and anxiety in an invisible marker versus a marker Augmented Reality system for the treatment of phobia towards small animals. International Journal of Human-Computer Studies. 69(6):440-453. doi:10.1016/j.ijhcs.2011.03.00244045369

SPATIO-TEMPORAL REGISTRATION IN AUGMENTED REALITY

The overarching goal of Augmented Reality (AR) is to provide users with the illusion that virtual and real objects coexist indistinguishably in the same space. An effective persistent illusion requires accurate registration between the real and the virtual objects, registration that is spatially and temporally coherent. However, visible misregistration can be caused by many inherent error sources, such as errors in calibration, tracking, and modeling, and system delay. This dissertation focuses on new methods that could be considered part of "the last mile" of spatio-temporal registration in AR: closed-loop spatial registration and low-latency temporal registration: 1. For spatial registration, the primary insight is that calibration, tracking and modeling are means to an end---the ultimate goal is registration. In this spirit I present a novel pixel-wise closed-loop registration approach that can automatically minimize registration errors using a reference model comprised of the real scene model and the desired virtual augmentations. Registration errors are minimized in both global world space via camera pose refinement, and local screen space via pixel-wise adjustments. This approach is presented in the context of Video See-Through AR (VST-AR) and projector-based Spatial AR (SAR), where registration results are measurable using a commodity color camera. 2. For temporal registration, the primary insight is that the real-virtual relationships are evolving throughout the tracking, rendering, scanout, and display steps, and registration can be improved by leveraging fine-grained processing and display mechanisms. In this spirit I introduce a general end-to-end system pipeline with low latency, and propose an algorithm for minimizing latency in displays (DLP DMD projectors in particular). This approach is presented in the context of Optical See-Through AR (OST-AR), where system delay is the most detrimental source of error. I also discuss future steps that may further improve spatio-temporal registration. Particularly, I discuss possibilities for using custom virtual or physical-virtual fiducials for closed-loop registration in SAR. The custom fiducials can be designed to elicit desirable optical signals that directly indicate any error in the relative pose between the physical and projected virtual objects.Doctor of Philosoph

A mixed reality framework for surgical navigation: approach and preliminary results

The overarching purpose of this research is to understand whether Mixed Reality can enhance a surgeon’s manipulations skills during minimally invasive procedures. Minimally-invasive surgery (MIS) utilizes small cuts in the skin - or sometimes natural orifices - to deploy instruments inside a patient’s body, while a live video feed of the surgical site is provided by an endoscopic camera and displayed on a screen. MIS is associated with many benefits: small scars, less pain and shorter hospitalization time as compared to traditional open surgery. However, these benefits come at a cost: because surgeons have to work by looking at a monitor, and not down on their own hands, MIS disrupts their eye-hand coordination and makes even simple surgical maneuvers challenging to perform. In this study, we wish to use Mixed Reality technology to superimpose anatomical models over the surgical site and explore if it can be used to mitigate this problem