Toward Real-Time Video-Enhanced Augmented Reality for Medical Visualization and Simulation

In this work we demonstrate two separate forms of augmented reality environments for use with minimally-invasive surgical techniques. In Chapter 2 it is demonstrated how a video feed from a webcam, which could mimic a laparoscopic or endoscopic camera used during an interventional procedure, can be used to identify the pose of the camera with respect to the viewed scene and augment the video feed with computer-generated information, such as rendering of internal anatomy not visible beyond the image surface, resulting in a simple augmented reality environment. Chapter 3 details our implementation of a similar system to the one previously mentioned, albeit with an external tracking system. Additionally, we discuss the challenges and considerations for expanding this system to support an external tracking system, specifically the Polaris Spectra optical tracker. Because of the relocation of the tracking origin to a point other than the camera center, there is an additional registration step necessary to establish the position of all components within the scene. This modification is expected to increase accuracy and robustness of the system

On uncertainty propagation in image-guided renal navigation: Exploring uncertainty reduction techniques through simulation and in vitro phantom evaluation

Image-guided interventions (IGIs) entail the use of imaging to augment or replace direct vision during therapeutic interventions, with the overall goal is to provide effective treatment in a less invasive manner, as an alternative to traditional open surgery, while reducing patient trauma and shortening the recovery time post-procedure. IGIs rely on pre-operative images, surgical tracking and localization systems, and intra-operative images to provide correct views of the surgical scene. Pre-operative images are used to generate patient-specific anatomical models that are then registered to the patient using the surgical tracking system, and often complemented with real-time, intra-operative images. IGI systems are subject to uncertainty from several sources, including surgical instrument tracking / localization uncertainty, model-to-patient registration uncertainty, user-induced navigation uncertainty, as well as the uncertainty associated with the calibration of various surgical instruments and intra-operative imaging devices (i.e., laparoscopic camera) instrumented with surgical tracking sensors. All these uncertainties impact the overall targeting accuracy, which represents the error associated with the navigation of a surgical instrument to a specific target to be treated under image guidance provided by the IGI system. Therefore, understanding the overall uncertainty of an IGI system is paramount to the overall outcome of the intervention, as procedure success entails achieving certain accuracy tolerances specific to individual procedures. This work has focused on studying the navigation uncertainty, along with techniques to reduce uncertainty, for an IGI platform dedicated to image-guided renal interventions. We constructed life-size replica patient-specific kidney models from pre-operative images using 3D printing and tissue emulating materials and conducted experiments to characterize the uncertainty of both optical and electromagnetic surgical tracking systems, the uncertainty associated with the virtual model-to-physical phantom registration, as well as the uncertainty associated with live augmented reality (AR) views of the surgical scene achieved by enhancing the pre-procedural model and tracked surgical instrument views with live video views acquires using a camera tracked in real time. To better understand the effects of the tracked instrument calibration, registration fiducial configuration, and tracked camera calibration on the overall navigation uncertainty, we conducted Monte Carlo simulations that enabled us to identify optimal configurations that were subsequently validated experimentally using patient-specific phantoms in the laboratory. To mitigate the inherent accuracy limitations associated with the pre-procedural model-to-patient registration and their effect on the overall navigation, we also demonstrated the use of tracked video imaging to update the registration, enabling us to restore targeting accuracy to within its acceptable range. Lastly, we conducted several validation experiments using patient-specific kidney emulating phantoms using post-procedure CT imaging as reference ground truth to assess the accuracy of AR-guided navigation in the context of in vitro renal interventions. This work helped find answers to key questions about uncertainty propagation in image-guided renal interventions and led to the development of key techniques and tools to help reduce optimize the overall navigation / targeting uncertainty

Essentials of Augmented Reality Software Development under Android Patform

Liitreaalsus on üha enam arenev tehnoloogia. Lisaks meelelahutuseleon liitreaalsus leidnud kasutust nii meditsiinis, sõjaväes, masinaehituses kui ka teistes suurtes ettevõtluse ning riigiga seotud valdkondades. Arendusmeeskondade eesmärk on saavutada võimalikult hea jõudlus ning visuaalsed tulemused nende poolt toodetavas tarkvaras sõltumata kasutuspiirkonnast. Liitreaalsuse tarkvara põhitehnoloogia sõltub vägapalju meeskonnale kättesaadavatest ressurssidest. See tähendab, et paremate võimalustega organisatsioonid saavad lubada endale tipptehnoloogiaid ning oma arendusmeeskondi, mille abil on neil võimalus implementeerida uusi liitreaalsuse tarkvaralahendusi. Samal ajal on aga tavalised firmad piiratud aja, meeskonna ja raha poolest, mis omakorda sunnib neid kasutama turul olemasolevaid lahendusi - tööriistakomplekte.Sellest lähtuvalt keskendub käesolev töö vajalikele teadmistele, mida läheb vaja erinevate liitreaalsuse tööriistakomplektide kasutamisel. Selleks, et luua edukalt valmis liitreaalsuse tarkvara, on välja valitud kindlad raamistikud, millest koostatakse ülevaade, mida testitakse ning võrreldakse. Lisaks sellele õpetatakse uurimise käigus selgeks ka mõned põhiteadmised liitreaalsuse arendamiseks Androidi platvormi näitel.Augmented Reality (AR) is an emerging technology. Besides entertainment, AR also is found to be used in medicine, military, engineering and other major fields of enterprise and government. Regardless of the application area, development teams usually target to achieve best performance and visual results in the AR software that they are providing. In addition, the core technology used behind a particular AR software depends a lot on resources available to the team. This means, that organizations with large resources can afford to implement AR software solutions using cutting-edge technologies build by their own engineering units, whereas ordinary companies are usually limited in time, staff and budget. Hence, forcing them to use existing market solutions - toolkits.From this perspective, this thesis work focuses on providing the basics of working with AR toolkits. In order to succeed in building an AR software, particular toolkits are selected to be reviewed, tested and compared. Moreover, during the investigation process some essentials of the AR development under Android platform are also studied

Desarrollo de una aplicación móvil mediante la integración de realidad aumentada y comunicación oral

This Bachelor project aims to create an augmented reality application which uses computer vision algorithms and technologies to provide assistance to the visually impaired in their daily tasks. The specific purpose of the application is to recognize scenes and objects after capturing them using the device’s native camera. Colour recognition is also present to aid in identifying objects and their properties such as their size and distance from the capturing device, such functionality may also help individuals which suffer from colour blindness. The application also interacts with the user by speech, effectively establishing a Speech Dialogue System (SDS). The general planning of this project was done considering the gained knowledge from project development courses done within the University of Carlos III, Madrid. In such, the required research and development was divided in a set number of stages, each with an independent number of tasks. This methodology has allowed us to define three major stages: Planning, Execution and Closing. A Work Breakdown Structure (WBS) has been used to make the structuring of the project’s planning easier. For the development of the application, JAVA programming language was used within the Android Studio development environment. OpenCV libraries were imported and used to implement the functionalities of the developed system. The final version of the application has thus used computer vision as a tool to provide additional information over real world scenes both with on screen representation and audible messages. As future work it would be interesting to reduce the computational load of computer vision algorithms. It would perhaps be convenient to further improve the OpenCV library and its recent API on Android.Ingeniería de Sistemas Audiovisuale

Using Auto-Ordering to Improve Object Transfer between Mobile Devices

People frequently form small groups in many social and professional situations: from conference attendees meeting at a coffee break, to siblings gathering at a family barbecue. These ad-hoc gatherings typically form into predictable geometries based on circles or circular arcs (called F-Formations). Because our lives are increasingly stored and represented by data on handheld devices, the desire to be able to share digital objects while in these groupings has increased. Using the relative position in these groups to facilitate file sharing could facilitate intuitive interfaces such as passing or flicking. However, there is no reliable, lightweight, ad-hoc technology for detecting and representing relative locations around a circle. In this thesis, we present three systems that can auto-order locations about a circle based on sensors standard on commodity smartphones. We tested two of these systems using an object passing task in a laboratory environment against unordered and proximity-based systems, and show that our techniques are faster, more accurate, and preferred by users

Multi-touch Detection and Semantic Response on Non-parametric Rear-projection Surfaces

The ability of human beings to physically touch our surroundings has had a profound impact on our daily lives. Young children learn to explore their world by touch; likewise, many simulation and training applications benefit from natural touch interactivity. As a result, modern interfaces supporting touch input are ubiquitous. Typically, such interfaces are implemented on integrated touch-display surfaces with simple geometry that can be mathematically parameterized, such as planar surfaces and spheres; for more complicated non-parametric surfaces, such parameterizations are not available. In this dissertation, we introduce a method for generalizable optical multi-touch detection and semantic response on uninstrumented non-parametric rear-projection surfaces using an infrared-light-based multi-camera multi-projector platform. In this paradigm, touch input allows users to manipulate complex virtual 3D content that is registered to and displayed on a physical 3D object. Detected touches trigger responses with specific semantic meaning in the context of the virtual content, such as animations or audio responses. The broad problem of touch detection and response can be decomposed into three major components: determining if a touch has occurred, determining where a detected touch has occurred, and determining how to respond to a detected touch. Our fundamental contribution is the design and implementation of a relational lookup table architecture that addresses these challenges through the encoding of coordinate relationships among the cameras, the projectors, the physical surface, and the virtual content. Detecting the presence of touch input primarily involves distinguishing between touches (actual contact events) and hovers (near-contact proximity events). We present and evaluate two algorithms for touch detection and localization utilizing the lookup table architecture. One of the algorithms, a bounded plane sweep, is additionally able to estimate hover-surface distances, which we explore for interactions above surfaces. The proposed method is designed to operate with low latency and to be generalizable. We demonstrate touch-based interactions on several physical parametric and non-parametric surfaces, and we evaluate both system accuracy and the accuracy of typical users in touching desired targets on these surfaces. In a formative human-subject study, we examine how touch interactions are used in the context of healthcare and present an exploratory application of this method in patient simulation. A second study highlights the advantages of touch input on content-matched physical surfaces achieved by the proposed approach, such as decreases in induced cognitive load, increases in system usability, and increases in user touch performance. In this experiment, novice users were nearly as accurate when touching targets on a 3D head-shaped surface as when touching targets on a flat surface, and their self-perception of their accuracy was higher

Realization Of A Spatial Augmented Reality System - A Digital Whiteboard Using a Kinect Sensor and a PC Projector

Recent rapid development of cost-effective, accurate digital imaging sensors, high-speed computational hardware, and tractable design software has given rise to the growing field of augmented reality in the computer vision realm. The system design of a 'Digital Whiteboard' system is presented with the intention of realizing a practical, cost-effective and publicly available spatial augmented reality system. A Microsoft Kinect sensor and a PC projector coupled with a desktop computer form a type of spatial augmented reality system that creates a projection based graphical user interface that can turn any wall or planar surface into a 'Digital Whiteboard'. The system supports two kinds of user inputs consisting of depth and infra-red information. An infra-red collimated light source, like that of a laser pointer pen, serves as a stylus for user input. The user can point and shine the infra-red stylus on the selected planar region and the reflection of the infra-red light source is registered by the system using the infra-red camera of the Kinect. Using the geometric transformation between the Kinect and the projector, obtained with system calibration, the projector displays contours corresponding to the movement of the stylus on the 'Digital Whiteboard' region, according to a smooth curve fitting algorithm. The described projector-based spatial augmented reality system provides new unique possibilities for user interaction with digital content

Tecniche per la rilevazione automatica marker-less di persone e marker-based di robot all'interno di reti di telecamere RGB-Depth

OpenPTrack is a state of the art solution for people detection and tracking, in this work we extended some of the functionalities (detection from highly tilted camera) of the software and introduced new ones (automatic ground plane equation calculator). Also, we test the feasibility and the behaviour of a mobile camera mounted on a people-following robot and dynamically registered in the OPT network through a fiducial cubic marke

Intraoperative Endoscopic Augmented Reality in Third Ventriculostomy

In neurosurgery, as a result of the brain-shift, the preoperative patient models used as a intraoperative reference change. A meaningful use of the preoperative virtual models during the operation requires for a model update. The NEAR project, Neuroendoscopy towards Augmented Reality, describes a new camera calibration model for high distorted lenses and introduces the concept of active endoscopes endowed with with navigation, camera calibration, augmented reality and triangulation modules