Adaptive User Perspective Rendering for Handheld Augmented Reality

Handheld Augmented Reality commonly implements some variant of magic lens rendering, which turns only a fraction of the user's real environment into AR while the rest of the environment remains unaffected. Since handheld AR devices are commonly equipped with video see-through capabilities, AR magic lens applications often suffer from spatial distortions, because the AR environment is presented from the perspective of the camera of the mobile device. Recent approaches counteract this distortion based on estimations of the user's head position, rendering the scene from the user's perspective. To this end, approaches usually apply face-tracking algorithms on the front camera of the mobile device. However, this demands high computational resources and therefore commonly affects the performance of the application beyond the already high computational load of AR applications. In this paper, we present a method to reduce the computational demands for user perspective rendering by applying lightweight optical flow tracking and an estimation of the user's motion before head tracking is started. We demonstrate the suitability of our approach for computationally limited mobile devices and we compare it to device perspective rendering, to head tracked user perspective rendering, as well as to fixed point of view user perspective rendering

Computational Re-Photography

Rephotographers aim to recapture an existing photograph from the same viewpoint. A historical photograph paired with a well-aligned modern rephotograph can serve as a remarkable visualization of the passage of time. However, the task of rephotography is tedious and often imprecise, because reproducing the viewpoint of the original photograph is challenging. The rephotographer must disambiguate between the six degrees of freedom of 3D translation and rotation, and the confounding similarity between the effects of camera zoom and dolly. We present a real-time estimation and visualization technique for rephotography that helps users reach a desired viewpoint during capture. The input to our technique is a reference image taken from the desired viewpoint. The user moves through the scene with a camera and follows our visualization to reach the desired viewpoint. We employ computer vision techniques to compute the relative viewpoint difference. We guide 3D movement using two 2D arrows. We demonstrate the success of our technique by rephotographing historical images and conducting user studies

Using mobile-based augmented reality and object detection for real-time Abalone growth monitoring

Abalone are becoming increasingly popular for human consumption. Whilst their popularity has risen, measuring the number and size distribution of Abalone at various stages of growth in existing farms remains a significant challenge. Current Abalone stock management techniques rely on manual inspection which is time consuming, causes stress to the animal, and results in mediocre data quality. To rectify this, we propose a novel mobile-based tool which combines object detection and augmented reality for the real-time counting and measuring of Abalone, that is both network and location independent. We applied our portable handset tool to both measure and count Abalone at various growth stages, and performed extended measuring evaluation to assess the robustness of our proposed approach. Our experimental results revealed that the proposed tool greatly outperforms traditional approaches and was able to successfully count up to 15 Abalone at various life stages with above 95% accuracy, as well as significantly decrease the time taken to measure Abalone while still maintaining an accuracy within a maximum error range of 2.5% of the Abalone’s actual size

Deep Thermal Imaging: Proximate Material Type Recognition in the Wild through Deep Learning of Spatial Surface Temperature Patterns

We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these textures into material types. This approach works effectively without the need for ambient light sources or direct contact with materials. Furthermore, the use of a deep learning network removes the need to handcraft the set of features for different materials. We evaluated the performance of the system by training it to recognise 32 material types in both indoor and outdoor environments. Our approach produced recognition accuracies above 98% in 14,860 images of 15 indoor materials and above 89% in 26,584 images of 17 outdoor materials. We conclude by discussing its potentials for real-time use in HCI applications and future directions.Comment: Proceedings of the 2018 CHI Conference on Human Factors in Computing System

Augmented Reality Framework and Demonstrator

Augmenting the real-world with digital information can improve the human perception in many ways. In recent years, a large amount of research has been conducted in the field of Augmented Reality (AR) and related technologies. Subsequently, different AR systems have been developed for the use in different areas such as medical, education, military, and entertainment. This thesis investigates augmented reality systems and challenges of realistic rendering in AR environment. Besides, an object-oriented framework, named ThirdEye, has been designed and implemented in order to facilitate the process of developing augmented reality applications for experimental purposes. This framework has been developed in two versions for desktop and mobile platforms. With ThirdEye, it is easier to port the same AR demo application to both platforms, manage and modify all AR demo application components, compared to the various existing libraries. Each feature that the ThirdEye framework includes, may be provided by other existing libraries separately but this framework provides those features in an easy-to-use manner. In order to evaluate usability and performance of ThirdEye and also for demonstrating challenges of simulating some of the light effects in the AR environment, such as shadow and refraction, several AR demos were developed using this framework. Performance of the implemented AR demos were benchmarked and bottlenecks of different components of the framework were investigated. This thesis explains the structure of the ThirdEye framework, its main components and the employed technologies and the Software Development Kits (SDKs). Furthermore, by using a simple demo, it is explained how this framework can be utilized to develop an AR application step by step. Lastly, several ideas for future development are described

Mental vision:a computer graphics platform for virtual reality, science and education

Despite the wide amount of computer graphics frameworks and solutions available for virtual reality, it is still difficult to find a perfect one fitting at the same time the many constraints of research and educational contexts. Advanced functionalities and user-friendliness, rendering speed and portability, or scalability and image quality are opposite characteristics rarely found into a same approach. Furthermore, fruition of virtual reality specific devices like CAVEs or wearable systems is limited by their costs and accessibility, being most of these innovations reserved to institutions and specialists able to afford and manage them through strong background knowledge in programming. Finally, computer graphics and virtual reality are a complex and difficult matter to learn, due to the heterogeneity of notions a developer needs to practice with before attempting to implement a full virtual environment. In this thesis we describe our contributions to these topics, assembled in what we called the Mental Vision platform. Mental Vision is a framework composed of three main entities. First, a teaching/research oriented graphics engine, simplifying access to 2D/3D real-time rendering on mobile devices, personal computers and CAVE systems. Second, a series of pedagogical modules to introduce and practice computer graphics and virtual reality techniques. Third, two advanced VR systems: a wearable, lightweight and handsfree mixed reality setup, and a four sides CAVE designed through off the shelf hardware. In this dissertation we explain our conceptual, architectural and technical approach, pointing out how we managed to create a robust and coherent solution reducing complexity related to cross-platform and multi-device 3D rendering, and answering simultaneously to contradictory common needs of computer graphics and virtual reality for researchers and students. A series of case studies evaluates how Mental Vision concretely satisfies these needs and achieves its goals on in vitro benchmarks and in vivo scientific and educational projects

Augmented reality X-ray vision on optical see-through head mounted displays

Abstract. In this thesis, we present the development and evaluation of an augmented reality X-ray system on optical see-through head-mounted displays. Augmented reality X-ray vision allows users to see through solid surfaces such as walls and facades, by augmenting the real view with virtual images representing the hidden objects. Our system is developed based on the optical see-through mixed reality headset Microsoft Hololens. We have developed an X-ray cutout algorithm that uses the geometric data of the environment and enables seeing through surfaces. We have developed four different visualizations as well based on the algorithm. The first visualization renders simply the X-ray cutout without displaying any information about the occluding surface. The other three visualizations display features extracted from the occluder surface to help the user to get better depth perception of the virtual objects. We have used Sobel edge detection to extract the information. The three visualizations differ in the way to render the extracted features. A subjective experiment is conducted to test and evaluate the visualizations and to compare them with each other. The experiment consists of two parts; depth estimation task and a questionnaire. Both the experiment and its results are presented in the thesis