Using CNNs For Users Segmentation In Video See-Through Augmented Virtuality

In this paper, we present preliminary results on the use of deep learning techniques to integrate the users self-body and other participants into a head-mounted video see-through augmented virtuality scenario. It has been previously shown that seeing users bodies in such simulations may improve the feeling of both self and social presence in the virtual environment, as well as user performance. We propose to use a convolutional neural network for real time semantic segmentation of users bodies in the stereoscopic RGB video streams acquired from the perspective of the user. We describe design issues as well as implementation details of the system and demonstrate the feasibility of using such neural networks for merging users bodies in an augmented virtuality simulation.Comment: 6 pages, 6 figures. Published in the 2nd International Conference on Artificial Intelligence & Virtual Reality (IEEE AIVR 2019

A window to your smartphone: exploring interaction and communication in immersive VR with augmented virtuality

A major drawback of most Head Mounted Displays (HMDs) used in immersive Virtual Reality (VR) is the visual and social isolation of users from their real-world surroundings while wearing these headsets. This partial isolation of users from the real-world might hinder social interactions with friends and family. To address this issue, we present a new method to allow people wearing VR HMDs to use their smartphones without removing their HMDs. To do this, we augment the scene inside the VR HMD with a view of the user's device so that the user can interact with the device without removing the headset. The idea involves the use of additional cameras, such as the Leap Motion device or a high-resolution RGB camera to capture the user's real-world surrounding and augment the virtual world with the content displayed on the smartphone screen. This setup allows VR users to have a window to their smartphone from within the virtual world and a�ord much of the functionality provided by their smartphones, with the potential to reduce undesirable visual and social isolation users may experience when using immersive VR HMDs. This work has been successfully submitted for presentation as a poster in the Computer and Robot Vision Conference 2017 in Edmonton, Alberta, and is scheduled to appear in the conference proceedings at the IEEE Xplore digital library later this year

Towards System Agnostic Calibration of Optical See-Through Head-Mounted Displays for Augmented Reality

This dissertation examines the developments and progress of spatial calibration procedures for Optical See-Through (OST) Head-Mounted Display (HMD) devices for visual Augmented Reality (AR) applications. Rapid developments in commercial AR systems have created an explosion of OST device options for not only research and industrial purposes, but also the consumer market as well. This expansion in hardware availability is equally matched by a need for intuitive standardized calibration procedures that are not only easily completed by novice users, but which are also readily applicable across the largest range of hardware options. This demand for robust uniform calibration schemes is the driving motive behind the original contributions offered within this work. A review of prior surveys and canonical description for AR and OST display developments is provided before narrowing the contextual scope to the research questions evolving within the calibration domain. Both established and state of the art calibration techniques and their general implementations are explored, along with prior user study assessments and the prevailing evaluation metrics and practices employed within. The original contributions begin with a user study evaluation comparing and contrasting the accuracy and precision of an established manual calibration method against a state of the art semi-automatic technique. This is the first formal evaluation of any non-manual approach and provides insight into the current usability limitations of present techniques and the complexities of next generation methods yet to be solved. The second study investigates the viability of a user-centric approach to OST HMD calibration through novel adaptation of manual calibration to consumer level hardware. Additional contributions describe the development of a complete demonstration application incorporating user-centric methods, a novel strategy for visualizing both calibration results and registration error from the user’s perspective, as well as a robust intuitive presentation style for binocular manual calibration. The final study provides further investigation into the accuracy differences observed between user-centric and environment-centric methodologies. The dissertation concludes with a summarization of the contribution outcomes and their impact on existing AR systems and research endeavors, as well as a short look ahead into future extensions and paths that continued calibration research should explore

Distributed Implementation of eXtended Reality Technologies over 5G Networks

Mención Internacional en el título de doctorThe revolution of Extended Reality (XR) has already started and is rapidly expanding as technology advances. Announcements such as Meta’s Metaverse have boosted the general interest in XR technologies, producing novel use cases. With the advent of the fifth generation of cellular networks (5G), XR technologies are expected to improve significantly by offloading heavy computational processes from the XR Head Mounted Display (HMD) to an edge server. XR offloading can rapidly boost XR technologies by considerably reducing the burden on the XR hardware, while improving the overall user experience by enabling smoother graphics and more realistic interactions. Overall, the combination of XR and 5G has the potential to revolutionize the way we interact with technology and experience the world around us. However, XR offloading is a complex task that requires state-of-the-art tools and solutions, as well as an advanced wireless network that can meet the demanding throughput, latency, and reliability requirements of XR. The definition of these requirements strongly depends on the use case and particular XR offloading implementations. Therefore, it is crucial to perform a thorough Key Performance Indicators (KPIs) analysis to ensure a successful design of any XR offloading solution. Additionally, distributed XR implementations can be intrincated systems with multiple processes running on different devices or virtual instances. All these agents must be well-handled and synchronized to achieve XR real-time requirements and ensure the expected user experience, guaranteeing a low processing overhead. XR offloading requires a carefully designed architecture which complies with the required KPIs while efficiently synchronizing and handling multiple heterogeneous devices. Offloading XR has become an essential use case for 5G and beyond 5G technologies. However, testing distributed XR implementations requires access to advanced 5G deployments that are often unavailable to most XR application developers. Conversely, the development of 5G technologies requires constant feedback from potential applications and use cases. Unfortunately, most 5G providers, engineers, or researchers lack access to cutting-edge XR hardware or applications, which can hinder the fast implementation and improvement of 5G’s most advanced features. Both technology fields require ongoing input and continuous development from each other to fully realize their potential. As a result, XR and 5G researchers and developers must have access to the necessary tools and knowledge to ensure the rapid and satisfactory development of both technology fields. In this thesis, we focus on these challenges providing knowledge, tools and solutiond towards the implementation of advanced offloading technologies, opening the door to more immersive, comfortable and accessible XR technologies. Our contributions to the field of XR offloading include a detailed study and description of the necessary network throughput and latency KPIs for XR offloading, an architecture for low latency XR offloading and our full end to end XR offloading implementation ready for a commercial XR HMD. Besides, we also present a set of tools which can facilitate the joint development of 5G networks and XR offloading technologies: our 5G RAN real-time emulator and a multi-scenario XR IP traffic dataset. Firstly, in this thesis, we thoroughly examine and explain the KPIs that are required to achieve the expected Quality of Experience (QoE) and enhanced immersiveness in XR offloading solutions. Our analysis focuses on individual XR algorithms, rather than potential use cases. Additionally, we provide an initial description of feasible 5G deployments that could fulfill some of the proposed KPIs for different offloading scenarios. We also present our low latency muti-modal XR offloading architecture, which has already been tested on a commercial XR device and advanced 5G deployments, such as millimeter-wave (mmW) technologies. Besides, we describe our full endto- end complex XR offloading system which relies on our offloading architecture to provide low latency communication between a commercial XR device and a server running a Machine Learning (ML) algorithm. To the best of our knowledge, this is one of the first successful XR offloading implementations for complex ML algorithms in a commercial device. With the goal of providing XR developers and researchers access to complex 5G deployments and accelerating the development of future XR technologies, we present FikoRE, our 5G RAN real-time emulator. FikoRE has been specifically designed not only to model the network with sufficient accuracy but also to support the emulation of a massive number of users and actual IP throughput. As FikoRE can handle actual IP traffic above 1 Gbps, it can directly be used to test distributed XR solutions. As we describe in the thesis, its emulation capabilities make FikoRE a potential candidate to become a reference testbed for distributed XR developers and researchers. Finally, we used our XR offloading tools to generate an XR IP traffic dataset which can accelerate the development of 5G technologies by providing a straightforward manner for testing novel 5G solutions using realistic XR data. This dataset is generated for two relevant XR offloading scenarios: split rendering, in which the rendering step is moved to an edge server, and heavy ML algorithm offloading. Besides, we derive the corresponding IP traffic models from the captured data, which can be used to generate realistic XR IP traffic. We also present the validation experiments performed on the derived models and their results.This work has received funding from the European Union (EU) Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie ETN TeamUp5G, grant agreement No. 813391.Programa de Doctorado en Multimedia y Comunicaciones por la Universidad Carlos III de Madrid y la Universidad Rey Juan CarlosPresidente: Narciso García Santos.- Secretario: Fernando Díaz de María.- Vocal: Aryan Kaushi

Stereoscopic Vision in Unmanned Aerial Vehicle Search and Rescue

Search and rescue operations are challenging due to the hazards imposed on the rescue teams. Team ARM IT has developed a virtual reality interface that controls a mounted camera payload on an unmanned aerial vehicle (UAV) through a head mounted display. This allows rescuers to manipulate a UAV to assist search and rescue missions safely and effectively through telepresence and enhanced situational awareness. The team tested these hypotheses by prototyping, testing, and refining individual components of the system through the use of flight simulation software and on-site volunteer testing. By providing a realistic sense of the UAV environment enhanced with relevant information, Team ARM IT’s project reduces the danger to the rescuers and provide cognitively natural situational awareness

3D Multi-user interactive visualization with a shared large-scale display

When the multiple users interact with a virtual environment on a largescale display there are several issues that need to be addressed to facilitate the interaction. In the thesis, three main topics for collaborative visualization are discussed; display setup, interactive visualization, and visual fatigue. The problems that the author is trying to address in this thesis are how multiple users can interact with a shared large-scale display depending on the display setups and how they can interact with the shared visualization in a way that doesn’t lead to visual fatigue. The first user study (Chapter 3) explores the display setups for multi-user interaction with a shared large-display. The author describes the design of the three main display setups (a shared view, a split screen, and a split screen with navigation information) and a demonstration using these setups. The user study found that the split screen and the split screen with navigation information can improve users’ confidence and reduce frustration level and are more preferred than a shared view. However, a shared view can still provide effective interaction and collaboration and the display setups cannot have a large impact on usability and workload. From the first study, the author employed a shared view for multi-user interactive visualization with a shared large-scale display due to the advantages of the shared view. To improve interactive visualization with a shared view for multiple users, the author designed and conducted the second user study (Chapter 4). A conventional interaction technique, the mean tracking method, was not effective for more than three users. In order to overcome the limitation of the current multi-user interactive visualization techniques, two interactive visualization techniques (the Object Shift Technique and Activity-based Weighted Mean Tracking method) were developed and were evaluated in the second user study. The Object Shift Technique translates the virtual objects in the opposite direction of movement of the Point of View (PoV) and the Activity-based Weighted Mean Tracking method assigns the higher weight to active users in comparison with stationary users to determine the location of the PoV. The results of the user study showed that these techniques can support collaboration, improve interactivity, and provide similar visual discomfort compared to the conventional method. The third study (Chapter 5) describes how to reduce visual fatigue for 3D stereoscopic visualization with a single point of view (PoV). When multiple users interact with 3D stereoscopic VR using multi-user interactive visualization techniques and they are close to the virtual objects, they can perceive 3D visual fatigue from the large disparity. To reduce the 3D visual fatigue, an Adaptive Interpupillary Distance (Adaptive IPD) adjustment technique was developed. To evaluate the Adaptive IPD method, the author compared to traditional 3D stereoscopic and the monoscopic visualization techniques. Through the user experiments, the author was able to confirm that the proposed method can reduce visual discomfort, yet maintain compelling depth perception as the result provided the most preferable 3D stereoscopic visualization experience. For these studies, the author developed a software framework and designed a set of experiments (Chapter 6). The framework architecture that contains the three main ideas are described. A demonstration application for multidimensional decision making was developed using the framework. The primary contributions of this thesis include a literature review of multiuser interaction with a shared large-scale display, deeper insights into three display setups for multi-user interaction, development of the Object Shift Techniques, the Activity-based Weighted Mean Tracking method, and the Adaptive Interpupillary Distance Adjustment technique, the evaluation of the three novel interaction techniques, development of a framework for supporting a multi-user interaction with a shared large-scale display and its application to multi-dimensional decision making VR system