Real-Time Occlusion Handling in Augmented Reality Based on an Object Tracking Approach

To produce a realistic augmentation in Augmented Reality, the correct relative positions of real objects and virtual objects are very important. In this paper, we propose a novel real-time occlusion handling method based on an object tracking approach. Our method is divided into three steps: selection of the occluding object, object tracking and occlusion handling. The user selects the occluding object using an interactive segmentation method. The contour of the selected object is then tracked in the subsequent frames in real-time. In the occlusion handling step, all the pixels on the tracked object are redrawn on the unprocessed augmented image to produce a new synthesized image in which the relative position between the real and virtual object is correct. The proposed method has several advantages. First, it is robust and stable, since it remains effective when the camera is moved through large changes of viewing angles and volumes or when the object and the background have similar colors. Second, it is fast, since the real object can be tracked in real-time. Last, a smoothing technique provides seamless merging between the augmented and virtual object. Several experiments are provided to validate the performance of the proposed method

Robust Estimation of Trifocal Tensors Using Natural Features for Augmented Reality Systems

Augmented reality deals with the problem of dynamically augmenting or enhancing the real world with computer generated virtual scenes. Registration is one of the most pivotal problems currently limiting AR applications. In this paper, a novel registration method using natural features based on online estimation of trifocal tensors is proposed. This method consists of two stages: offline initialization and online registration. Initialization involves specifying four points in two reference images respectively to build the world coordinate system on which a virtual object will be augmented. In online registration, the natural feature correspondences detected from the reference views are tracked in the current frame to build the feature triples. Then these triples are used to estimate the corresponding trifocal tensors in the image sequence by which the four specified points are transferred to compute the registration matrix for augmentation. The estimated registration matrix will be used as an initial estimate for a nonlinear optimization method that minimizes the actual residual errors based on the Levenberg-Marquardt (LM) minimization method, thus making the results more robust and stable. This paper also proposes a robust method for estimating the trifocal tensors, where a modified RANSAC algorithm is used to remove outliers. Compared with standard RANSAC, our method can significantly reduce computation complexity, while overcoming the disturbance of mismatches. Some experiments have been carried out to demonstrate the validity of the proposed approach

Near-Field Depth Perception in See-Through Augmented Reality

This research studied egocentric depth perception in an augmented reality (AR) environment. Specifically, it involved measuring depth perception in the near visual field by using quantitative methods to measure the depth relationships between real and virtual objects. This research involved two goals; first, engineering a depth perception measurement apparatus and related calibration andmeasuring techniques for collecting depth judgments, and second, testing its effectiveness by conducting an experiment. The experiment compared two complimentary depth judgment protocols: perceptual matching (a closed-loop task) and blind reaching (an open-loop task). It also studied the effect of a highly salient occluding surface; this surface appeared behind, coincident with, and in front of virtual objects. Finally, the experiment studied the relationship between dark vergence and depth perception

Augmentoidun todellisuuden lääketieteelliset sovellukset

Insinöörityössä käsiteltiin augmentoidun todellisuuden lääketieteellisiä sovelluksia. Työn tavoitteena oli esittää mahdollisimman laaja katsaus augmentoituun todellisuuteen ja erityisesti sen hyödyntämismahdollisuuksiin lääketieteen alalla, koska aikaisempia suomenkielisiä katsauksia käsiteltyyn aiheeseen liittyen ei ole juurikaan julkaistu. Alkuosassa määriteltiin augmentoitu todellisuus ja kerrottiin sen historiasta. Sovellusalueet, joissa augmentoitua todellisuutta on yleensä käytetty, esiteltiin kattavasti. Teknologia-osiossa kerrottiin erilaisista tekniikoista, joilla augmentoitu todellisuus on tehty mahdolliseksi. Näyttöteknologiat, heijastettavan informaation kohdentaminen sekä tietokoneen ja käyttäjän välinen vuorovaikutus käsiteltiin. Lääketieteen alueella tutkittiin augmentoidun todellisuuden hyödyntämistä leikkauksissa, harjoittelun apuvälineenä, terapiassa ja potilaiden kuntoutuksessa, sekä esiteltiin uusimpia lääketieteellisiä sovelluksia mobiililaitteille. Lopuksi pohdittiin augmentoidun todellisuuden mahdollisia tulevaisuuden sovelluksia.This thesis covers augmented reality applications in medical field. The aim was to present a comprehensive review to augmented reality and especially its utilization possibilities in the medical field, because hardly any previous publications written in Finnish have not been released covering similar kind of topics. In the first part of the thesis the augmented reality is defined and its history is introduced. Application fields, where augmented reality is mostly used, are widely presented. In the technology section, different techniques that make augmented reality possible are explained. Display technologies, tracking the superimposed information, as well as the interaction between computer and the user are clarified. Utilization possibilities in medical field were studied in surgeries, in medical student’s training aid, in therapy and in patient rehabilitation. The newest medical applications in mobile devices are presented. At the end of the thesis possible future applications in augmented reality are deliberated

Integration intuitiver Eingabegeräte ins MITK zur Optimierung von Mensch-Computer-Interaktion in der Medizin

Der Einsatz von virtuellen Szenen in der Medizin gewinnt zunehmend an Bedeutung, weil Navigations- und Planungshilfen für den Arzt geschaffen werden. Die Verwendung von komplexen Computersimulationen soll den Behandlungsprozess verkürzen und gleichzeitig die Behandlungsqualität durch eine gezielte Vorgehensweise verbessern. Wegen der hohen Komplexität von Bildverarbeitungsalgorithmen ist es jedoch schwierig, eine einfache Interaktion mit den medizinischen Daten zu ermöglichen. Zudem werden Benutzereingaben durch die Eingabegeräte wie Maus und Tastatur eingeschränkt, da diese in vielen Fällen keine einfache Steuerung zulassen. Diese Diplomarbeit beschäftigt sich mit der Umsetzung intuitiver Interaktionskonzepte für den alltäglichen klinischen Gebrauch durch Verwendung von intuitiven Eingabegeräten (3D Maus, Wii Controller). Die Entwicklungen basieren auf dem Medical Imaging Interaction Toolkit (MITK) des Deutschen Krebsforschungszentrums (DKFZ). Dabei wurden von einer einfachen Kamerafahrt in einer Volumenvisualisierung über die Realisierung eines Headtracking in einer virtuellen Realität bis hin zu der Interaktion mit 3D Objekten konkrete Anwendungsbeispiele erarbeitet, analysiert und bewertet

Optimised calibration, registration and tracking for image enhanced surgical navigation in ENT operations

EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Ανάπτυξη τεχνολογιών επαυξημένης πραγματικότητας στην ιατρική εκπαίδευση με προσομοιωτές

Στην παρούσα διδακτορική διατριβή παρουσιάζουμε ένα πρωτοπόρο σύστημα εκπαίδευσης και αξιολόγησης βασικών δεξιοτήτων λαπαροσκοπικής χειρουργικής σε περιβάλλον Επαυξημένης Πραγματικότητας (ΕΠ). Το προτεινόμενο σύστημα αποτελεί μια πλήρως λειτουργική πλατφόρμα εκπαίδευσης η οποία επιτρέπει σε χειρουργούς να εξασκηθούν χρησιμοποιώντας πραγματικά λαπαροσκοπικά εργαλεία και αλληλεπιδρώντας με ψηφιακά αντικείμενα εντός ενός πραγματικού περιβάλλοντος εκπαίδευσης. Το σύστημα αποτελείται από ένα τυπικό κουτί λαπαροσκοπικής εκπαίδευσης, πραγματικά χειρουργικά εργαλεία, κάμερα και συστοιχία αισθητήρων που επιτρέπουν την ανίχνευση και καταγραφή των κινήσεων του χειρουργού σε πραγματικό χρόνο. Χρησιμοποιώντας το προτεινόμενο σύστημα, σχεδιάσαμε και υλοποιήσαμε σενάρια εκπαίδευσης παρόμοια με τις ασκήσεις του προγράμματος FLS®, στοχεύοντας σε δεξιότητες όπως η αίσθηση βάθους, ο συντονισμός χεριού-ματιού, και η παράλληλη χρήση δύο χεριών. Επιπλέον των βασικών δεξιοτήτων, το προτεινόμενο σύστημα χρησιμοποιήθηκε για τον σχεδιασμό σεναρίου εξάσκησης διαδικαστικών δεξιοτήτων, οι οποίες περιλάμβανουν την εφαρμογή χειρουργικών clips καθώς και την απολίνωση εικονικής αρτηρίας, σε περιβάλλον ΕΠ. Τα αποτελέσματα συγκριτικών μελετών μεταξύ έμπειρων και αρχαρίων χειρουργών που πραγματοποιήθηκαν στα πλαίσια της παρούσας διατριβής υποδηλώνουν την εγκυρότητα του προτεινόμενου συστήματος. Επιπλέον, εξήχθησαν σημαντικά συμπεράσματα σχετικά με την πιθανή χρήση της ΕΑ στην λαπαροσκοπική προσομοίωση. Η συγκεκριμένη τεχνολογία προσφέρει αυξημένη αίσθηση οπτικού ρεαλισμού και ευελιξία στον σχεδιασμό εκπαιδευτικών σεναρίων, παρουσιάζοντας σημαντικά μικρότερες απαιτήσεις από πλευράς εξοπλισμού σε σύγκριση με τις υπάρχουσες εμπορικές πλατφόρμες. Βάσει των αποτελεσμάτων της παρούσας διατριβής μπορεί με ασφάλεια να εξαχθεί το συμπέρασμα πως η ΕΠ αποτελεί μια πολλά υποσχόμενη τεχνολογία που θα μπορούσε να χρησιμοποιηθεί για τον σχεδιασμό προσομοιωτών λαπαροσκοπικής χειρουργικής ως εναλλακτική των υπαρχόντων τεχνολογιών και συστημάτων.In this thesis we present what is, to the best of our knowledge, the first framework for training and assessment of fundamental psychomotor and procedural laparoscopic skills in an interactive Augmented Reality (AR) environment. The proposed system is a fully-featured laparoscopic training platform, allowing surgeons to practice by manipulating real instruments while interacting with virtual objects within a real environment. It consists of a standard laparoscopic box-trainer, real instruments, a camera and a set of sensory devices for real-time tracking of surgeons’ actions. The proposed framework has been used for the implementation of AR-based training scenarios similar to the drills of the FLS® program, focusing on fundamental laparoscopic skills such as depth-perception, hand-eye coordination and bimanual operation. Moreover, this framework allowed the implementation of a proof-of-concept procedural skills training scenario, which involved clipping and cutting of a virtual artery within an AR environment. Comparison studies conducted for the evaluation of the presented framework indicated high content and face validity. In addition, significant conclusions regarding the potentials of introducing AR in laparoscopic simulation training and assessment were drawn. This technology provides an advanced sense of visual realism combined with a great flexibility in training task prototyping, with minimum requirements in terms of hardware as compared to commercially available platforms. Thereby, it can be safely stated that AR is a promising technology which can indeed provide a valuable alternative to the training modalities currently used in MIS

Image Guidance in Telemanipulator Assisted Urology Surgery

This thesis outlines the development of an image guided surgery system, intended for use in \davinci assisted radical prostatectomy but more generally applicable to laparoscopic urology surgery. We defined the key performance parameter of the system as the accuracy of overlaying modelled anatomy onto the surgical scene. This thesis is primarily concerned with determining the system accuracy based on an analysis of the system's components. A common error measure was defined for all system components. This is an on screen error (measured in pixels) based on the error in projecting a single point lying near the apex of the prostate with the endoscope in a typical surgical pose. In this case the projected point was approximately 200 mm from the endoscope lens. An intraoperative coordinate system is first defined as the coordinate system of an optical tracking system used to track the endoscope. The MRI image of the patient is transformed into the intraoperative coordinate system. Prior to surgery the endoscope is calibrated and during surgery the endoscope is tracked, defining a transform from the coordinates of the optical tracking system to the endoscope screen. This transform is used to project the MRI image onto the endoscope video display. The early part of the thesis describes a novel algorithm for registering MRI to ultrasound images of the bone which was used to put the MRI image into the intraoperative coordinate system. Using this algorithm avoids the need for fiducial markers. The table below shows the errors (as on screen pixel RMS) due to using this algorithm. An approximate value as RMS distance error at the prostate apex point is also included

Virtual Reality and Augmented Reality in Digestive Surgery

International audienceno abstrac