Using CNNs For Users Segmentation In Video See-Through Augmented Virtuality

In this paper, we present preliminary results on the use of deep learning techniques to integrate the users self-body and other participants into a head-mounted video see-through augmented virtuality scenario. It has been previously shown that seeing users bodies in such simulations may improve the feeling of both self and social presence in the virtual environment, as well as user performance. We propose to use a convolutional neural network for real time semantic segmentation of users bodies in the stereoscopic RGB video streams acquired from the perspective of the user. We describe design issues as well as implementation details of the system and demonstrate the feasibility of using such neural networks for merging users bodies in an augmented virtuality simulation.Comment: 6 pages, 6 figures. Published in the 2nd International Conference on Artificial Intelligence & Virtual Reality (IEEE AIVR 2019

The Weird Giggle: Attending to Affect in Virtual Reality

Virtual Reality (VR) is once again causing a stir, with conflicting assertions over its potential to usher in a glorious posthuman phase of freedom or to immerse bodies wearing headsets in pure and meaningless violence. This paper integrates philosophies of affect and affective experiences in VR by means of a practical application of phenomenological reflection. The combination of phenomenology and affect is valuable for articulating the lived experience of something unprecedented or disorienting, and for expanding the language of critique. The practical affective experiences of VR are from one particular VR artwork: MAN A VR by Gibson / Martelli, which uses captured data from dancers performing the dance improvisation form Skinner Releasing Technique (SRT) to animate the figures the VR world. SRT is also the movement practice facilitating philosophical reflections on the experience of being in the VR world. In this paper, passages directly describing moments of experience in MAN A VR extracted directly from research journals act as affective counterpoints to the theoretical discussion. The result is an expansion of the somatic register of VR, at the same time as a grounding of concepts from affect theory within contemporary digital culture

You Can't Hide Behind Your Headset: User Profiling in Augmented and Virtual Reality

Virtual and Augmented Reality (VR, AR) are increasingly gaining traction thanks to their technical advancement and the need for remote connections, recently accentuated by the pandemic. Remote surgery, telerobotics, and virtual offices are only some examples of their successes. As users interact with VR/AR, they generate extensive behavioral data usually leveraged for measuring human behavior. However, little is known about how this data can be used for other purposes. In this work, we demonstrate the feasibility of user profiling in two different use-cases of virtual technologies: AR everyday application ($N=34$) and VR robot teleoperation ($N=35$). Specifically, we leverage machine learning to identify users and infer their individual attributes (i.e., age, gender). By monitoring users' head, controller, and eye movements, we investigate the ease of profiling on several tasks (e.g., walking, looking, typing) under different mental loads. Our contribution gives significant insights into user profiling in virtual environments

Virtual Reality and Its Application in Education

Virtual reality is a set of technologies that enables two-way communication, from computer to user and vice versa. In one direction, technologies are used to synthesize visual, auditory, tactile, and sometimes other sensory experiences in order to provide the illusion that practically non-existent things can be seen, heard, touched, or otherwise felt. In the other direction, technologies are used to adequately record human movements, sounds, or other potential input data that computers can process and use. This book contains six chapters that cover topics including definitions and principles of VR, devices, educational design principles for effective use of VR, technology education, and use of VR in technical and natural sciences

Enhancing interaction in mixed reality

With continuous technological innovation, we observe mixed reality emerging from research labs into the mainstream. The arrival of capable mixed reality devices transforms how we are entertained, consume information, and interact with computing systems, with the most recent being able to present synthesized stimuli to any of the human senses and substantially blur the boundaries between the real and virtual worlds. In order to build expressive and practical mixed reality experiences, designers, developers, and stakeholders need to understand and meet its upcoming challenges. This research contributes a novel taxonomy for categorizing mixed reality experiences and guidelines for designing mixed reality experiences. We present the results of seven studies examining the challenges and opportunities of mixed reality experiences, the impact of modalities and interaction techniques on the user experience, and how to enhance the experiences. We begin with a study determining user attitudes towards mixed reality in domestic and educational environments, followed by six research probes that each investigate an aspect of reality or virtuality. In the first, a levitating steerable projector enables us to investigate how the real world can be enhanced without instrumenting the user. We show that the presentation of in-situ instructions for navigational tasks leads to a significantly higher ability to observe and recall real-world landmarks. With the second probe, we enhance the perception of reality by superimposing information usually not visible to the human eye. In amplifying the human vision, we enable users to perceive thermal radiation visually. Further, we examine the effect of substituting physical components with non-functional tangible proxies or entirely virtual representations. With the third research probe, we explore how to enhance virtuality to enable a user to input text on a physical keyboard while being immersed in the virtual world. Our prototype tracked the user’s hands and keyboard to enable generic text input. Our analysis of text entry performance showed the importance and effect of different hand representations. We then investigate how to touch virtuality by simulating generic haptic feedback for virtual reality and show how tactile feedback through quadcopters can significantly increase the sense of presence. Our final research probe investigates the usability and input space of smartphones within mixed reality environments, pairing the user’s smartphone as an input device with a secondary physical screen. Based on our learnings from these individual research probes, we developed a novel taxonomy for categorizing mixed reality experiences and guidelines for designing mixed reality experiences. The taxonomy is based on the human sensory system and human capabilities of articulation. We showcased its versatility and set our research probes into perspective by organizing them inside the taxonomic space. The design guidelines are divided into user-centered and technology-centered. It is our hope that these will contribute to the bright future of mixed reality systems while emphasizing the new underlining interaction paradigm.Mixed Reality (vermischte Realitäten) gehen aufgrund kontinuierlicher technologischer Innovationen langsam von der reinen Forschung in den Massenmarkt über. Mit der Einführung von leistungsfähigen Mixed-Reality-Geräten verändert sich die Art und Weise, wie wir Unterhaltungsmedien und Informationen konsumieren und wie wir mit Computersystemen interagieren. Verschiedene existierende Geräte sind in der Lage, jeden der menschlichen Sinne mit synthetischen Reizen zu stimulieren. Hierdurch verschwimmt zunehmend die Grenze zwischen der realen und der virtuellen Welt. Um eindrucksstarke und praktische Mixed-Reality-Erfahrungen zu kreieren, müssen Designer und Entwicklerinnen die künftigen Herausforderungen und neuen Möglichkeiten verstehen. In dieser Dissertation präsentieren wir eine neue Taxonomie zur Kategorisierung von Mixed-Reality-Erfahrungen sowie Richtlinien für die Gestaltung von solchen. Wir stellen die Ergebnisse von sieben Studien vor, in denen die Herausforderungen und Chancen von Mixed-Reality-Erfahrungen, die Auswirkungen von Modalitäten und Interaktionstechniken auf die Benutzererfahrung und die Möglichkeiten zur Verbesserung dieser Erfahrungen untersucht werden. Wir beginnen mit einer Studie, in der die Haltung der nutzenden Person gegenüber Mixed Reality in häuslichen und Bildungsumgebungen analysiert wird. In sechs weiteren Fallstudien wird jeweils ein Aspekt der Realität oder Virtualität untersucht. In der ersten Fallstudie wird mithilfe eines schwebenden und steuerbaren Projektors untersucht, wie die Wahrnehmung der realen Welt erweitert werden kann, ohne dabei die Person mit Technologie auszustatten. Wir zeigen, dass die Darstellung von in-situ-Anweisungen für Navigationsaufgaben zu einer deutlich höheren Fähigkeit führt, Sehenswürdigkeiten der realen Welt zu beobachten und wiederzufinden. In der zweiten Fallstudie erweitern wir die Wahrnehmung der Realität durch Überlagerung von Echtzeitinformationen, die für das menschliche Auge normalerweise unsichtbar sind. Durch die Erweiterung des menschlichen Sehvermögens ermöglichen wir den Anwender:innen, Wärmestrahlung visuell wahrzunehmen. Darüber hinaus untersuchen wir, wie sich das Ersetzen von physischen Komponenten durch nicht funktionale, aber greifbare Replikate oder durch die vollständig virtuelle Darstellung auswirkt. In der dritten Fallstudie untersuchen wir, wie virtuelle Realitäten verbessert werden können, damit eine Person, die in der virtuellen Welt verweilt, Text auf einer physischen Tastatur eingeben kann. Unser Versuchsdemonstrator detektiert die Hände und die Tastatur, zeigt diese in der vermischen Realität an und ermöglicht somit die verbesserte Texteingaben. Unsere Analyse der Texteingabequalität zeigte die Wichtigkeit und Wirkung verschiedener Handdarstellungen. Anschließend untersuchen wir, wie man Virtualität berühren kann, indem wir generisches haptisches Feedback für virtuelle Realitäten simulieren. Wir zeigen, wie Quadrokopter taktiles Feedback ermöglichen und dadurch das Präsenzgefühl deutlich steigern können. Unsere letzte Fallstudie untersucht die Benutzerfreundlichkeit und den Eingaberaum von Smartphones in Mixed-Reality-Umgebungen. Hierbei wird das Smartphone der Person als Eingabegerät mit einem sekundären physischen Bildschirm verbunden, um die Ein- und Ausgabemodalitäten zu erweitern. Basierend auf unseren Erkenntnissen aus den einzelnen Fallstudien haben wir eine neuartige Taxonomie zur Kategorisierung von Mixed-Reality-Erfahrungen sowie Richtlinien für die Gestaltung von solchen entwickelt. Die Taxonomie basiert auf dem menschlichen Sinnessystem und den Artikulationsfähigkeiten. Wir stellen die vielseitige Verwendbarkeit vor und setzen unsere Fallstudien in Kontext, indem wir sie innerhalb des taxonomischen Raums einordnen. Die Gestaltungsrichtlinien sind in nutzerzentrierte und technologiezentrierte Richtlinien unterteilt. Es ist unsere Anliegen, dass diese Gestaltungsrichtlinien zu einer erfolgreichen Zukunft von Mixed-Reality-Systemen beitragen und gleichzeitig die neuen Interaktionsparadigmen hervorheben

A Digital Future in Virtual Reality — Insights for Training

Virtual reality is now being used throughout various sectors. It is a tool which is being increasingly relied upon to support cost-effective and safe opportunities to build skills development. There has, however, been little research into whether a virtual environment provides the same effectiveness as a real-world environment. For virtual reality to be an effective tool, we must better understand the impact of using it. To determine this, we investigate whether there is an additional cognitive load when operating in a virtual environment and we measure whether such a load impacts upon an individual’s performance. Through the use of a ‘quadrant’ study in both real and virtual environments and with both the presence and absence of a secondary task, we identified that there is no significant cognitive load added when working within the virtual environment, and so the use of virtual reality can indeed be effective in terms of comparative performance with the real-world. This research was conducted with approval of the Human Research Ethics Committee (Tasmania) Network; the reference number for the study is: H0018156

A window to your smartphone: exploring interaction and communication in immersive VR with augmented virtuality

A major drawback of most Head Mounted Displays (HMDs) used in immersive Virtual Reality (VR) is the visual and social isolation of users from their real-world surroundings while wearing these headsets. This partial isolation of users from the real-world might hinder social interactions with friends and family. To address this issue, we present a new method to allow people wearing VR HMDs to use their smartphones without removing their HMDs. To do this, we augment the scene inside the VR HMD with a view of the user's device so that the user can interact with the device without removing the headset. The idea involves the use of additional cameras, such as the Leap Motion device or a high-resolution RGB camera to capture the user's real-world surrounding and augment the virtual world with the content displayed on the smartphone screen. This setup allows VR users to have a window to their smartphone from within the virtual world and a�ord much of the functionality provided by their smartphones, with the potential to reduce undesirable visual and social isolation users may experience when using immersive VR HMDs. This work has been successfully submitted for presentation as a poster in the Computer and Robot Vision Conference 2017 in Edmonton, Alberta, and is scheduled to appear in the conference proceedings at the IEEE Xplore digital library later this year