The sense of embodiment in Virtual Reality and its assessment methods

The sense of embodiment refers to the sensations of being inside, having, and controlling a body. In virtual reality, it is possible to substitute a person’s body with a virtual body, referred to as an avatar. Modulations of the sense of embodiment through modifications of this avatar have perceptual and behavioural consequences on users that can influence the way users interact with the virtual environment. Therefore, it is essential to define metrics that enable a reliable assessment of the sense of embodiment in virtual reality to better understand its dimensions, the way they interact, and their influence on the quality of interaction in the virtual environment. In this review, we first introduce the current knowledge on the sense of embodiment, its dimensions (senses of agency, body ownership, and self-location), and how they relate the ones with the others. Then, we dive into the different methods currently used to assess the sense of embodiment, ranging from questionnaires to neurophysiological measures. We provide a critical analysis of the existing metrics, discussing their advantages and drawbacks in the context of virtual reality. Notably, we argue that real-time measures of embodiment, which are also specific and do not require double tasking, are the most relevant in the context of virtual reality. Electroencephalography seems a good candidate for the future if its drawbacks (such as its sensitivity to movement and practicality) are improved. While the perfect metric has yet to be identified if it exists, this work provides clues on which metric to choose depending on the context, which should hopefully contribute to better assessing and understanding the sense of embodiment in virtual reality

Gaze and Peripheral Vision Analysis for Human-Environment Interaction: Applications in Automotive and Mixed-Reality Scenarios

This thesis studies eye-based user interfaces which integrate information about the user’s perceptual focus-of-attention into multimodal systems to enrich the interaction with the surrounding environment. We examine two new modalities: gaze input and output in the peripheral field of view. All modalities are considered in the whole spectrum of the mixed-reality continuum. We show the added value of these new forms of multimodal interaction in two important application domains: Automotive User Interfaces and Human-Robot Collaboration. We present experiments that analyze gaze under various conditions and help to design a 3D model for peripheral vision. Furthermore, this work presents several new algorithms for eye-based interaction, like deictic reference in mobile scenarios, for non-intrusive user identification, or exploiting the peripheral field view for advanced multimodal presentations. These algorithms have been integrated into a number of software tools for eye-based interaction, which are used to implement 15 use cases for intelligent environment applications. These use cases cover a wide spectrum of applications, from spatial interactions with a rapidly changing environment from within a moving vehicle, to mixed-reality interaction between teams of human and robots.In dieser Arbeit werden blickbasierte Benutzerschnittstellen untersucht, die Infor- mationen ¨uber das Blickfeld des Benutzers in multimodale Systeme integrieren, um neuartige Interaktionen mit der Umgebung zu erm¨oglichen. Wir untersuchen zwei neue Modalit¨aten: Blickeingabe und Ausgaben im peripheren Sichtfeld. Alle Modalit¨aten werden im gesamten Spektrum des Mixed-Reality-Kontinuums betra- chtet. Wir zeigen die Anwendung dieser neuen Formen der multimodalen Interak- tion in zwei wichtigen Dom¨anen auf: Fahrerassistenzsysteme und Werkerassistenz bei Mensch-Roboter-Kollaboration. Wir pr¨asentieren Experimente, die blickbasierte Benutzereingaben unter verschiedenen Bedingungen analysieren und helfen, ein 3D- Modell f¨ur das periphere Sehen zu entwerfen. Dar¨uber hinaus stellt diese Arbeit mehrere neue Algorithmen f¨ur die blickbasierte Interaktion vor, wie die deiktis- che Referenz in mobilen Szenarien, die nicht-intrusive Benutzeridentifikation, oder die Nutzung des peripheren Sichtfeldes f¨ur neuartige multimodale Pr¨asentationen. Diese Algorithmen sind in eine Reihe von Software-Werkzeuge integriert, mit de- nen 15 Anwendungsf¨alle f¨ur intelligente Umgebungen implementiert wurden. Diese Demonstratoren decken ein breites Anwendungsspektrum ab: von der r¨aumlichen In- teraktionen aus einem fahrenden Auto heraus bis hin zu Mixed-Reality-Interaktionen zwischen Mensch-Roboter-Teams

Immersive virtual reality methods in cognitive neuroscience and neuropsychology: the Virtual Reality Everyday Assessment Lab (VR-EAL).an immersive neuropsychological test battery of everyday cognitive functions

In cognitive neuroscience and neuropsychology, the collection of cognitive and behavioural data is predominantly achieved by implementing paper-and-pencil and computerized (i.e., 2D and 3D applications) assessments. However, these psychometric tools in clinics and/or laboratories display several limitations and discrepancies between the observed performance in the laboratory/clinic and the actual performance of individuals in everyday life. The functional and predictive association between an individual's performance on a set of neuropsychological tests and the individual's performance in various everyday life settings is called ecological validity. Ecological validity is considered an important issue that cannot be resolved by the currently available assessment tools. Virtual reality head-mounted displays (HMD) appear to be effective research tools, which may address the problem of ecological validity in neuropsychological testing. However, their widespread implementation is hindered by virtual reality induced symptoms and effects (VRISE) and the lack of skills in virtual reality software development. In this PhD, a technological systematic literature review of the reasons for adverse symptomatology was conducted and suggestions and technological knowledge for the implementation of virtual reality HMD systems in cognitive neuroscience provided. The review indicated features pertinent to display, sound, motion tracking, navigation, ergonomic interactions, user experience, and computer hardware that should be considered by researchers. Subsequently, a meta-analysis of 44 neuroscientific or neuropsychological studies involving virtual reality HMD systems was performed. The meta-analysis of the virtual reality studies demonstrated that new generation HMDs induce significantly less VRISE and marginally fewer dropouts. Importantly, the commercial versions of the new generation HMDs with ergonomic interactions had zero incidents of adverse symptomatology and dropouts. HMDs equivalent to or greater than the commercial versions of contemporary HMDs accompanied with ergonomic interactions are suitable for implementation in cognitive neuroscience. Another aim of this PhD was to devise a brief tool to appraise and report both the quality of software features and VRISE intensity quantitatively; such a tool does not currently exist. The Virtual Reality Neuroscience Questionnaire (VRNQ; Kourtesis et al., 2019) was developed to assess the quality of virtual reality software in terms of user experience, game mechanics, in-game assistance, and VRISE. Forty participants aged between 28 and 43 years were recruited (18 gamers and 22 non-gamers) for the study. They participated in 3 different virtual reality sessions until they felt weary or discomfort and subsequently filled in the VRNQ. The results demonstrated that VRNQ is a valid tool for assessing virtual reality software as it has good convergent, discriminant, and construct validity. The maximum duration of virtual reality sessions should be between 55 and 70 min when the virtual reality software meets or exceeds the parsimonious cut-offs of the VRNQ, and the users are familiarized with the virtual reality system. Also, gaming experience does not affect how long virtual reality sessions should last. Furthermore, while the quality of virtual reality software substantially modulates the maximum duration of virtual reality sessions, age and education do not. Finally, deeper immersion, better quality of graphics and sound, and more helpful in-game instructions and prompts were found to reduce VRISE intensity. The VRNQ facilitates the brief assessment and reporting of the quality of virtual reality software features and/or the intensity of VRISE, while its minimum and parsimonious cut-offs may appraise the suitability of virtual reality software for implementation in research and clinical settings. However, the development of virtual reality software is predominantly dependent on third parties (e.g., freelancers or companies) with programming and software development skills. A solution that will promote the adoption of immersive virtual reality as a research and clinical tool might be the in-house development of virtual reality research/clinical software by computer science literate cognitive scientists or research software engineers. In Chapter 4, guidelines are offered for the development of virtual reality software in cognitive neuroscience and neuropsychology, by describing and discussing the stages of the development of Virtual Reality Everyday Assessment Lab (VR-EAL), the first neuropsychological battery in immersive virtual reality. Techniques for evaluating cognitive functions within a realistic storyline are discussed. The utility of various assets in Unity, software development kits, and other software are described so that cognitive scientists can overcome challenges pertinent to VRISE and the quality of the virtual reality software. In addition, VR-EAL is evaluated in accordance with the necessary criteria for virtual reality software for research purposes. The virtual reality neuroscience questionnaire (VRNQ) was implemented to appraise the quality of the three versions of VR-EAL in terms of user experience, game mechanics, in-game assistance, and VRISE. Twenty-five participants aged between 20 and 45 years with 12–16 years of full-time education evaluated various versions of VR-EAL. The final version of VR-EAL achieved high scores in every sub-score of the VRNQ and exceeded its parsimonious cut-offs. It also appeared to have better in-game assistance and game mechanics, while its improved graphics substantially increased the quality of the user experience and almost eradicated VRISE. The results substantially support the feasibility of the development of effective virtual reality research and clinical software without the presence of VRISE during a 60-min virtual reality session. In Chapter 5, validation of VR-EAL as an assessment of prospective memory, episodic memory, attention, and executive functions using an ecologically valid approach is examined. Performance on the VR-EAL, an immersive virtual reality neuropsychological battery, is examined against an extensive paper-and-pencil neuropsychological battery. Forty-one participants (21 females) were recruited: 18 gamers and 23 non-gamers who attended both an immersive virtual reality and a paper-and-pencil testing session. Bayesian Pearson correlation analyses were conducted to assess construct and convergent validity of the VR-EAL. Bayesian t-tests were performed to compare virtual reality and paper-and-pencil testing in terms of administration time, similarity to real life tasks (i.e., ecological validity), and pleasantness. VR-EAL scores were significantly correlated with their equivalent scores on the paper-and-pencil tests. The participants’ reports indicated that the VR-EAL tasks were considered significantly more ecologically valid and pleasant than the paper-and-pencil neuropsychological battery. The VR-EAL battery also had a shorter administration time. The VR-EAL appears to be an effective neuropsychological tool for the assessment of everyday cognitive functions, and has enhanced ecological validity, a highly pleasant testing experience, and does not induce cybersickness. In the final part of this thesis, the preparatory attentional and memory (PAM) and the multiprocess theories of prospective memory are examined by attempting to identify the cognitive functions which may predict the individual’s performance on ecologically valid prospective memory tasks in the same group of participants described in Chapter 5. Bayesian t-tests were conducted to explore the differences among different prospective memory tasks (e.g., event-based and time-based) and prospective memory tasks with varying delays between encoding and the recall of the intended action (e.g., short-delay versus long-delay). Bayesian linear regression analyses were performed to examine the predictors of VR-EAL scores. The results revealed that the type of prospective memory task does not play a significant role in everyday prospective memory functioning, but instead the length of delay between encoding and retrieving the prospective memory intention plays a central role. Support for the PAM and MP frameworks was found in non-focal and focal event-based tasks respectively. However, the findings, inferring a dynamic interplay between automatic and intentional monitoring and retrieval processes, agree with the inclusive approach of the multiprocess framework. Also, the role of executive functions appears crucial in everyday PM. Finally, everyday PM is predominantly facilitated by episodic memory, visuospatial attention, auditory attention, and executive functions. In conclusion, this PhD thesis attempted to show how immersive virtual reality research methods may be implemented efficiently without the confounding effect of cybersickness symptomatology in order to enhance the ecological validity of neuropsychological testing and contribute to our understanding of everyday cognitive ability

Implications of the uncanny valley of avatars and virtual characters for human-computer interaction

Technological innovations made it possible to create more and more realistic figures. Such figures are often created according to human appearance and behavior allowing interaction with artificial systems in a natural and familiar way. In 1970, the Japanese roboticist Masahiro Mori observed, however, that robots and prostheses with a very - but not perfect - human-like appearance can elicit eerie, uncomfortable, and even repulsive feelings. While real people or stylized figures do not seem to evoke such negative feelings, human depictions with only minor imperfections fall into the "uncanny valley," as Mori put it. Today, further innovations in computer graphics led virtual characters into the uncanny valley. Thus, they have been subject of a number of disciplines. For research, virtual characters created by computer graphics are particularly interesting as they are easy to manipulate and, thus, can significantly contribute to a better understanding of the uncanny valley and human perception. For designers and developers of virtual characters such as in animated movies or games, it is important to understand how the appearance and human-likeness or virtual realism influence the experience and interaction of the user and how they can create believable and acceptable avatars and virtual characters despite the uncanny valley. This work investigates these aspects and is the next step in the exploration of the uncanny valley. This dissertation presents the results of nine studies examining the effects of the uncanny valley on human perception, how it affects interaction with computing systems, which cognitive processes are involved, and which causes may be responsible for the phenomenon. Furthermore, we examine not only methods for avoiding uncanny or unpleasant effects but also the preferred characteristics of virtual faces. We bring the uncanny valley into context with related phenomena causing similar effects. By exploring the eeriness of virtual animals, we found evidence that the uncanny valley is not only related to the dimension of human-likeness, which significantly change our view on the phenomenon. Furthermore, using advanced hand tracking and virtual reality technologies, we discovered that avatar realism is connected to other factors, which are related to the uncanny valley and depend on avatar realism. Affinity with the virtual ego and the feeling of presence in the virtual world were also affected by gender and deviating body structures such as a reduced number of fingers. Considering the performance while typing on keyboards in virtual reality, we also found that the perception of the own avatar depends on the user's individual task proficiencies. This thesis concludes with implications that not only extends existing knowledge about virtual characters, avatars and the uncanny valley but also provide new design guidelines for human-computer interaction and virtual reality

Proceedings of the 9th Arab Society for Computer Aided Architectural Design (ASCAAD) international conference 2021 (ASCAAD 2021): architecture in the age of disruptive technologies: transformation and challenges.

The ASCAAD 2021 conference theme is Architecture in the age of disruptive technologies: transformation and challenges. The theme addresses the gradual shift in computational design from prototypical morphogenetic-centered associations in the architectural discourse. This imminent shift of focus is increasingly stirring a debate in the architectural community and is provoking a much needed critical questioning of the role of computation in architecture as a sole embodiment and enactment of technical dimensions, into one that rather deliberately pursues and embraces the humanities as an ultimate aspiration