Off-Line Camera-Based Calibration for Optical See-Through Head-Mounted Displays

In recent years, the entry into the market of self contained optical see-through headsets with integrated multi-sensor capabilities has led the way to innovative and technology driven augmented reality applications and has encouraged the adoption of these devices also across highly challenging medical and industrial settings. Despite this, the display calibration process of consumer level systems is still sub-optimal, particularly for those applications that require high accuracy in the spatial alignment between computer generated elements and a real-world scene. State-of-the-art manual and automated calibration procedures designed to estimate all the projection parameters are too complex for real application cases outside laboratory environments. This paper describes an off-line fast calibration procedure that only requires a camera to observe a planar pattern displayed on the see-through display. The camera that replaces the user’s eye must be placed within the eye-motion-box of the see-through display. The method exploits standard camera calibration and computer vision techniques to estimate the projection parameters of the display model for a generic position of the camera. At execution time, the projection parameters can then be refined through a planar homography that encapsulates the shift and scaling effect associated with the estimated relative translation from the old camera position to the current user’s eye position. Compared to classical SPAAM techniques that still rely on the human element and to other camera based calibration procedures, the proposed technique is flexible and easy to replicate in both laboratory environments and real-world settings

Обеспечение визуальной когерентности в обучающих системах дополненной реальности с учетом авиакосмической специфики

In May 2022, Saudi Arabian Military Industries, a Saudi government agency, acquired an augmented reality training platform for pilots. In September, the Boeing Corporation began the development of an augmented reality pilot simulator. In November, a similar project was launched by BAE Systems, a leading British developer of aeronautical engineering. These facts allow us to confidently speak about the beginning of a new era of aviation simulators – simulators using the augmented reality technology. One of the promising advantages of this technology is the ability to safely simulate dangerous situations in the real world. A necessary condition for using this advantage is to ensure the visual coherence of augmented reality scenes: virtual objects must be indistinguishable from real ones. All the global IT leaders consider augmented reality as the subsequent surge of radical changes in digital electronics, so visual coherence is becoming a key issue for the future of IT, and in aerospace applications, visual coherence has already acquired practical significance. The Russian Federation lags far behind in studying the problems of visual coherence in general and for augmented reality flight simulators in particular: at the time of publication the authors managed to find only two papers on the subject in the Russian research space, while abroad their number is already approximately a thousand. The purpose of this review article is to create conditions for solving the problem. Visual coherence depends on many factors: lighting, color tone, shadows from virtual objects on real ones, mutual reflections, textures of virtual surfaces, optical aberrations, convergence and accommodation, etc. The article reviews the publications devoted to methods for assessing the conditions of illumination and color tone of a real scene and transferring them to virtual objects using various probes and by individual images, as well as by rendering virtual objects in augmented reality scenes, using neural networks.В мае 2022 года саудовская правительственная структура Saudi Arabian Military Industries приобрела обучающую платформу дополненной реальности для летчиков, в сентябре корпорация Boeing начала разработку тренажера пилота дополненной реальности, в ноябре стартовал аналогичный проект ведущего британского разработчика авиационной техники BAE Systems. Эти факты позволяют уверенно говорить о начале новой эпохи авиационных тренажеров – тренажеров с применением технологии дополненной реальности. Одно из перспективных преимуществ данной технологии – возможность безопасного моделирования опасных ситуаций в реальном мире. Необходимым условием использования этого преимущества является обеспечение визуальной когерентности сцен дополненной реальности: виртуальные объекты должны быть неотличимы от реальных. Все мировые IT-лидеры рассматривают дополненную реальность как следующую «большую волну» радикальных изменений в цифровой электронике, поэтому визуальная когерентность становится ключевым вопросом для будущего IT, а в аэрокосмических приложениях визуальная когерентность уже приобрела практическое значение. В РФ имеет место серьезное отставание в изучении проблематики визуальной когерентности в целом и для авиатренажеров дополненной реальности в частности: на момент публикации авторам удалось обнаружить в российском научном пространстве только две работы по теме, тогда как за рубежом их число уже около тысячи. Цель настоящей обзорной статьи – создать условия для купирования проблемы. Визуальная когерентность зависит от многих факторов: освещения, цветового тона, теней от виртуальных объектов на реальных, взаимных отражений, текстур виртуальных поверхностей, оптических аберраций, конвергенции и аккомодации и др. В статье анализируются публикации, посвященные методам оценки условий освещенности и цветового тона реальной сцены и переноса таковых на виртуальные объекты с использованием зондов и по отдельным изображениям, а также по рендерингу виртуальных объектов в сценах дополненной реальности, в том числе с применением нейросетей

Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings

In this paper, we address the challenge of estimating the 6DoF pose of objects in 2D equirectangular images. This solution allows the transition to the objects’ 3D model from their current pose. In particular, it finds application in the educational use of 360° videos, where it enhances the learning experience of students by making it more engaging and immersive due to the possible interaction with 3D virtual models. We developed a general approach usable for any object and shape. The only requirement is to have an accurate CAD model, even without textures of the item, whose pose must be estimated. The developed pipeline has two main steps: vehicle segmentation from the image background and estimation of the vehicle pose. To accomplish the first task, we used deep learning methods, while for the second, we developed a 360° camera simulator in Unity to generate synthetic equirectangular images used for comparison. We conducted our tests using a miniature truck model whose CAD was at our disposal. The developed algorithm was tested using a metrological analysis applied to real data. The results showed a mean difference of 1.5° with a standard deviation of 1° from the ground truth data for rotations, and 1.4 cm with a standard deviation of 1.5 cm for translations over a research range of ±20° and ±20 cm, respectively

A Systematic Literature Review of Augmented Reality in Engineering Education : Hardware, Software, Student Motivation & Development Recommendations.

Augmented Reality (AR) is a technology that has benefited from the massification of computational devices, putting it in the focus of researchers as a novel teaching aid in engineering. For this very reason, a great amount of information about AR in engineering education is emerging constantly. To synthesize the information, a Systematic Literature Review (SLR) was carried out and 4 research questions were raised. It was found out the researcher´s trend for the development and testing of software that takes advantage of AR for engineering students. It was also found that Germany and India are the leading nations on investigations about the investigated topic. On the software side, Unity is the most used tool for creating applications and that the target hardware is smartphone. Finally, the high interest of researchers to increase motivation in students is evidenced in this SLR and recommendations were made based on the researcher’s findings.La Realidad Aumentada (AR) es una tecnología que se ha beneficiado de la masificación de los dispositivos computacionales, poniéndola en el foco de los investigadores como una novedosa ayuda didáctica en ingeniería. Por esta razón, surge constantemente una gran cantidad de información sobre la AR en la educación en ingeniería. Para sintetizar la información se realizó una Revisión Sistemática de Literatura (SLR) y se plantearon 4 preguntas de investigación. Se encontró la tendencia de los investigadoradores para el desarrollo y testeode software que aprovecha la AR para estudiantes de ingeniería. También se encontró que Alemania e India son las naciones líderes en investigaciones sobre el tema investigado. En el lado del software, Unity es la herramienta más utilizada para crear aplicaciones y que el hardware de destino es el teléfono inteligente. Finalmente, el alto interés de los investigadores por aumentar la motivación en los estudiantes se evidencia en esta SLR y se hicieron recomendaciones basadas en los hallazgos reportados en cada documento analizado

Augmented Reality and Health Informatics: A Study based on Bibliometric and Content Analysis of Scholarly Communication and Social Media

Healthcare outcomes have been shown to improve when technology is used as part of patient care. Health Informatics (HI) is a multidisciplinary study of the design, development, adoption, and application of IT-based innovations in healthcare services delivery, management, and planning. Augmented Reality (AR) is an emerging technology that enhances the user’s perception and interaction with the real world. This study aims to illuminate the intersection of the field of AR and HI. The domains of AR and HI by themselves are areas of significant research. However, there is a scarcity of research on augmented reality as it applies to health informatics. Given both scholarly research and social media communication having contributed to the domains of AR and HI, research methodologies of bibliometric and content analysis on scholarly research and social media communication were employed to investigate the salient features and research fronts of the field. The study used Scopus data (7360 scholarly publications) to identify the bibliometric features and to perform content analysis of the identified research. The Altmetric database (an aggregator of data sources) was used to determine the social media communication for this field. The findings from this study included Publication Volumes, Top Authors, Affiliations, Subject Areas and Geographical Locations from scholarly publications as well as from a social media perspective. The highest cited 200 documents were used to determine the research fronts in scholarly publications. Content Analysis techniques were employed on the publication abstracts as a secondary technique to determine the research themes of the field. The study found the research frontiers in the scholarly communication included emerging AR technologies such as tracking and computer vision along with Surgical and Learning applications. There was a commonality between social media and scholarly communication themes from an applications perspective. In addition, social media themes included applications of AR in Healthcare Delivery, Clinical Studies and Mental Disorders. Europe as a geographic region dominates the research field with 50% of the articles and North America and Asia tie for second with 20% each. Publication volumes show a steep upward slope indicating continued research. Social Media communication is still in its infancy in terms of data extraction, however aggregators like Altmetric are helping to enhance the outcomes. The findings from the study revealed that the frontier research in AR has made an impact in the surgical and learning applications of HI and has the potential for other applications as new technologies are adopted

Context-Dependent Information Elements in the Car: Explorative Analysis of Static and Dynamic Head-Up-Displays

Head-up-displays (HUDs) illustrate a particular static number of information elements in the driver’s primary field of view. Since the display can obscure the reality, a dynamic HUD presents context-dependent information elements. To become familiar with a user-optimal number of information elements and its essential information elements, we conducted a user study with n = 183 participants. We focused the context on an urban, a rural and a highway trip. Afterwards, a within-subject experiment using a high-fidelity driving simulator (n = 27) reveals the following: Dynamic HUDs significantly lower the average over speeding by 3.45 km/h compared to static HUDs. This speed above the speed limit equals 15.33% of the average speed in urban areas. Steering angle and speed can capture the context. Practitioners can use these findings to decrease the number of information elements in HUDs, thereby possibly increasing traffic safety

Untersuchung des Zusammenhangs von Presence und User Experience in der Virtuellen Realität als Grundlage für die Entwicklung von Trainingssystemen und Medizinprodukten in der Chirurgie

Die Dissertation untersucht grundlegende psychologische Faktoren für die Nutzbarmachung der Potentiale von Virtueller Realität für die chirurgische Ausbildung und Medizinprodukteentwicklung.:I Abkürzungsverzeichnis .................................................................................................................... 4 II Abbildungsverzeichnis..................................................................................................................... 5 1 Einführung in die Thematik ............................................................................................................. 6 1.1 Einsatz von Virtual Reality in Medizin und Chirurgie .............................................................. 6 1.1.1 Virtual Reality für chirurgisches Training und Telechirurgie ........................................... 6 1.1.2 Virtual Reality in der Therapie und Rehabilitation.......................................................... 7 1.2 Presence, User Experience und Usability ................................................................................ 8 1.2.1 Begriffe ............................................................................................................................ 8 1.2.2 Verwendete Messmethoden für Presence, User Experience und Usability ................. 10 1.3 Potentiale von Virtual Reality in der Medizinproduktentwicklung....................................... 11 1.4 Motivation und Ziele der Arbeit............................................................................................ 13 1.5 Experimenteller Aufbau ........................................................................................................ 16 1.5.1 Studienaufbau ............................................................................................................... 16 1.5.2 Experimentaltechnik...................................................................................................... 20 2 Publikationsmanuskripte............................................................................................................... 22 2.1 Brade, J., Lorenz, M. et al. (2017). Being there again – Presence in real and virtual environments and its relation to usability and user experience using a mobile navigation task..... 22 2.2 Lorenz, M. et al. (2018). Presence and User Experience in a Virtual Environment under the Influence of Ethanol: An Explorative Study....................................................................................... 35 2.3 Diskussion.............................................................................................................................. 51 3 Zusammenfassung der Arbeit ....................................................................................................... 53 3.1 Hintergrund ........................................................................................................................... 53 3.2 Ziele ....................................................................................................................................... 53 3.3 Methoden.............................................................................................................................. 54 3.4 Ergebnisse.............................................................................................................................. 54 3.5 Schlussfolgerungen................................................................................................................ 55 4 Literaturverzeichnis....................................................................................................................... 56 III Darstellung des eigenen Beitrags.................................................................................................. 64 IV Erklärung über die eigenständige Abfassung der Arbeit............................................................... 66 V Wissenschaftliche Veröffentlichungen und Vorträge ................................................................... 67 VI Danksagung ................................................................................................................................... 7

Perspective Preserving Solution for Quasi-Orthoscopic Video See-Through HMDs

In non-orthoscopic video see-through (VST) head-mounted displays (HMDs), depth perception through stereopsis is adversely affected by sources of spatial perception errors. Solutions for parallax-free and orthoscopic VST HMDs were considered to ensure proper space perception but at expenses of an increased bulkiness and weight. In this work, we present a hybrid video-optical see-through HMD the geometry of which explicitly violates the rigorous conditions of orthostereoscopy. For properly recovering natural stereo fusion of the scene within the personal space in a region around a predefined distance from the observer, we partially resolve the eye-camera parallax by warping the camera images through a perspective preserving homography that accounts for the geometry of the VST HMD and refers to such distance. For validating our solution; we conducted objective and subjective tests. The goal of the tests was to assess the efficacy of our solution in recovering natural depth perception in the space around said reference distance. The results obtained showed that the quasi-orthoscopic setting of the HMD; together with the perspective preserving image warping; allow the recovering of a correct perception of the relative depths. The perceived distortion of space around the reference plane proved to be not as severe as predicted by the mathematical models

Current challenges and future research directions in augmented reality for education

The progression and adoption of innovative learning methodologies signify that a respective part of society is open to new technologies and ideas and thus is advancing. The latest innovation in teaching is the use of Augmented Reality (AR). Applications using this technology have been deployed successfully in STEM (Science, Technology, Engineering, and Mathematics) education for delivering the practical and creative parts of teaching. Since AR technology already has a large volume of published studies about education that reports advantages, limitations, effectiveness, and challenges, classifying these projects will allow for a review of the success in the different educational settings and discover current challenges and future research areas. Due to COVID-19, the landscape of technology-enhanced learning has shifted more toward blended learning, personalized learning spaces and user-centered approach with safety measures. The main findings of this paper include a review of the current literature, investigating the challenges, identifying future research areas, and finally, reporting on the development of two case studies that can highlight the first steps needed to address these research areas. The result of this research ultimately details the research gap required to facilitate real-time touchless hand interaction, kinesthetic learning, and machine learning agents with a remote learning pedagogy

PassWalk: Spatial Authentication Leveraging Lateral Shift and Gaze on Mobile Headsets

Peer reviewe