Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing

[EN] Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which allows researchers to simulate environments in controlled laboratory conditions with high levels of sense of presence and interactivity, is becoming more popular in emotion research. Moreover, its synergy with implicit measurements and machine-learning techniques has the potential to impact transversely in many research areas, opening new opportunities for the scientific community. This paper presents a systematic review of the emotion recognition research undertaken with physiological and behavioural measures using head-mounted displays as elicitation devices. The results highlight the evolution of the field, give a clear perspective using aggregated analysis, reveal the current open issues and provide guidelines for future research.This research was funded by European Commission, grant number H2020-825585 HELIOS.Marín-Morales, J.; Llinares Millán, MDC.; Guixeres Provinciale, J.; Alcañiz Raya, ML. (2020). Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing. Sensors. 20(18):1-26. https://doi.org/10.3390/s20185163S126201

Aerospace medicine and biology: A continuing bibliography with indexes (supplement 359)

This bibliography lists 164 reports, articles and other documents introduced into the NASA Scientific and Technical Information System during Jan. 1992. Subject coverage includes: aerospace medicine and physiology, life support systems and man/system technology, protective clothing, exobiology and extraterrestrial life, planetary biology, and flight crew behavior and performance

FACING EXPERIENCE: A PAINTER’S CANVAS IN VIRTUAL REALITY

Full version unavailable due to 3rd party copyright restrictions.This research investigates how shifts in perception might be brought about through the development of visual imagery created by the use of virtual environment technology. Through a discussion of historical uses of immersion in art, this thesis will explore how immersion functions and why immersion has been a goal for artists throughout history. It begins with a discussion of ancient cave drawings and the relevance of Plato’s Allegory of the Cave. Next it examines the biological origins of “making special.” The research will discuss how this concept, combined with the ideas of “action” and “reaction,” has reinforced the view that art is fundamentally experiential rather than static. The research emphasizes how present-day virtual environment art, in providing a space that engages visitors in computer graphics, expands on previous immersive artistic practices. The thesis examines the technical context in which the research occurs by briefly describing the use of computer science technologies, the fundamentals of visual arts practices, and the importance of aesthetics in new media and provides a description of my artistic practice. The aim is to investigate how combining these approaches can enhance virtual environments as artworks. The computer science of virtual environments includes both hardware and software programming. The resultant virtual environment experiences are technologically dependent on the types of visual displays being used, including screens and monitors, and their subsequent viewing affordances. Virtual environments fill the field of view and can be experienced with a head mounted display (HMD) or a large screen display. The sense of immersion gained through the experience depends on how tracking devices and related peripheral devices are used to facilitate interaction. The thesis discusses visual arts practices with a focus on how illusions shift our cognition and perception in the visual modalities. This discussion includes how perceptual thinking is the foundation of art experiences, how analogies are the foundation of cognitive experiences and how the two intertwine in art experiences for virtual environments. An examination of the aesthetic strategies used by artists and new media critics are presented to discuss new media art. This thesis investigates the visual elements used in virtual environments and prescribes strategies for creating art for virtual environments. Methods constituting a unique virtual environment practice that focuses on visual analogies are discussed. The artistic practice that is discussed as the basis for this research also concentrates on experiential moments and shifts in perception and cognition and references Douglas Hofstadter, Rudolf Arnheim and John Dewey. iv Virtual environments provide for experiences in which the imagery generated updates in real time. Following an analysis of existing artwork and critical writing relative to the field, the process of inquiry has required the creation of artworks that involve tracking systems, projection displays, sound work, and an understanding of the importance of the visitor. In practice, the research has shown that the visitor should be seen as an interlocutor, interacting from a first-person perspective with virtual environment events, where avatars or other instrumental intermediaries, such as guns, vehicles, or menu systems, do not to occlude the view. The aesthetic outcomes of this research are the result of combining visual analogies, real time interactive animation, and operatic performance in immersive space. The environments designed in this research were informed initially by paintings created with imagery generated in a hypnopompic state or during the moments of transitioning from sleeping to waking. The drawings often emphasize emotional moments as caricatures and/or elements of the face as seen from a number of perspectives simultaneously, in the way of some cartoons, primitive artwork or Cubist imagery. In the imagery, the faces indicate situations, emotions and confrontations which can offer moments of humour and reflective exploration. At times, the faces usurp the space and stand in representation as both face and figure. The power of the placement of the caricatures in the paintings become apparent as the imagery stages the expressive moment. The placement of faces sets the scene, establishes relationships and promotes the honesty and emotions that develop over time as the paintings are scrutinized. The development process of creating virtual environment imagery starts with hand drawn sketches of characters, develops further as paintings on “digital canvas”, are built as animated, three-dimensional models and finally incorporated into a virtual environment. The imagery is generated while drawing, typically with paper and pencil, in a stream of consciousness during the hypnopompic state. This method became an aesthetic strategy for producing a snappy straightforward sketch. The sketches are explored further as they are worked up as paintings. During the painting process, the figures become fleshed out and their placement on the page, in essence brings them to life. These characters inhabit a world that I explore even further by building them into three dimensional models and placing them in computer generated virtual environments. The methodology of developing and placing the faces/figures became an operational strategy for building virtual environments. In order to open up the range of art virtual environments, and develop operational strategies for visitors’ experience, the characters and their facial features are used as navigational strategies, signposts and methods of wayfinding in order to sustain a stream of consciousness type of navigation. Faces and characters were designed to represent those intimate moments of self-reflection and confrontation that occur daily within ourselves and with others. They sought to reflect moments of wonderment, hurt, curiosity and humour that could subsequently be relinquished for more practical or purposeful endeavours. They were intended to create conditions in which visitors might reflect upon their emotional state, v enabling their understanding and trust of their personal space, in which decisions are made and the nature of world is determined. In order to extend the split-second, frozen moment of recognition that a painting affords, the caricatures and their scenes are given new dimensions as they become characters in a performative virtual reality. Emotables, distinct from avatars, are characters confronting visitors in the virtual environment to engage them in an interactive, stream of consciousness, non-linear dialogue. Visitors are also situated with a role in a virtual world, where they were required to adapt to the language of the environment in order to progress through the dynamics of a drama. The research showed that imagery created in a context of whimsy and fantasy could bring ontological meaning and aesthetic experience into the interactive environment, such that emotables or facially expressive computer graphic characters could be seen as another brushstroke in painting a world of virtual reality

Sonic Interactions in Virtual Environments

This open access book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

Enhanced attention using head-mounted virtual reality

Some evidence suggests that experiencing a given scenario using virtual reality (VR) may engage greater attentional resources than experiencing the same scenario on a 2D computer monitor. However, the underlying neural processes associated with these VR-related effects, especially those pertaining to current consumer-friendly head-mounted displays of virtual reality (HMD-VR), remain unclear. Here, two experiments were conducted to compare task performance and EEG-based neural metrics captured during a perceptual discrimination task presented on two different viewing platforms. Forty participants (20–25 years old) completed this task using both an HMD-VR and traditional computer monitor in a within-group, randomized design. Although Experiment I (n = 20) was solely behavioral in design, Experiment II (n = 20) utilized combined EEG recordings to interrogate the neural correlates underlying potential performance differences across platforms. These experiments revealed that (1) there was no significant difference in the amount of arousal measured between platforms and (2) selective attention abilities in HMD-VR environment were enhanced from both a behavioral and neural perspective. These findings suggest that the allocation of attentional resources in HMD-VR may be superior to approaches more typically used to assess these abilities (e.g., desktop/laptop/tablet computers with 2D screens)

Les apports de la réalité virtuelle et de la vidéo-oculographie à l’évaluation des intérêts sexuels

La pléthysmographie pénienne est reconnue comme la méthode la plus rigoureuse pour l’évaluation des intérêts sexuels. Cet instrument pourrait cependant bénéficier d’amélioration au niveau de sa validité écologique et de sa vulnérabilité aux tentatives de falsification. Cet outil est également confronté aux contraintes éthiques et légales liées à la présentation de stimuli produits à partir de modèle humain pour susciter l’excitation sexuelle. La présente thèse doctorale propose de pallier aux lacunes de la procédure pléthysmographique en la combinant à des stimuli générés par ordinateur présentés en immersion virtuelle et à la vidéo-oculographie. Dans le cadre d’une première étude empirique, il s’agit de déterminer si l’usage de stimuli générés par ordinateur présentés en immersion virtuelle est préférable à la modalité auditive actuellement utilisée. Les réponses érectiles d’agresseurs sexuels d’enfants et d’hommes sans paraphilie connue sont enregistrées et la capacité des deux modalités à produire des profils d’excitation sexuels et des indices de déviance propres aux intérêts sexuels est comparée. Dans le cadre d’une seconde étude expérimentale, les mouvements oculaires sont analysés afin de détecter les tentatives de contrôle volontaire de la réponse érectile lors d’une procédure d’évaluation pléthysmographique. Les réponses érectiles et les mouvements oculaires d’hommes sans dossier judiciaire sont enregistrés lors du visionnement de stimuli sous trois différentes conditions, dont une tâche de suppression de la réponse érectile. Dans l’ensemble, les résultats démontrent que les stimuli générés par ordinateur présentés en immersion virtuelle génèrent des profils d’excitation sexuelle ainsi que des indices de déviance ayant une précision de classification et de discrimination significativement supérieure à la modalité auditive. D’autre part, les résultats soulignent la capacité des mouvements oculaires à identifier une signature oculaire propre à l’utilisation d’une stratégie cognitive d’inhibition de la réponse érectile qui se caractérise notamment par la mise à l’écart du contenu sexuel et le ralentissement global du processus d’exploration visuel. Ce projet de recherche souligne les avantages inhérents à la présentation en immersion virtuelle de stimuli générés par ordinateur ainsi qu’à l’étude des mouvements oculaires lors de l’évaluation pléthysmographique et vient appuyer l’implantation progressive de cette modalité dans les milieux d’évaluations cliniques.Penile plethysmography is considered to be the most rigorous method for sexual interest assessment. The instrument could however benefit from improvements in terms of its ecological validity and vulnerability to faking attempts. This tool also faces ethical and legal constraints related to the presentation of stimuli depicting human subjects to entice sexual arousal. This doctoral thesis seeks to address penile plethysmography’s shortcomings by combining it with computer-generated stimuli presented in virtual immersion as well as eye-tracking devices. A first empirical study was conducted to determine whether the presentation of computer-generated stimuli in virtual immersion is preferable to the auditory modality currently being used. Erectile responses from child molesters and men without sexual deviances were recorded and the ability of both modalities to generate sexual arousal profiles and deviance differentials indicative of sexual interests were compared. In a second experimental study, eye-movements were explored in an attempt to identify the presence of cognitive strategies responsible for erectile inhibition during the plethysmographic procedure. Erectile responses and eye-movements of men without criminal records were recorded when viewing stimuli under three different conditions, including an erectile inhibition task. Overall, results show that the presentation of computer-generated stimuli in virtual immersion generates sexual arousal profiles and deviance differentials, which allow significantly higher classification accuracy and discriminant performance than the auditory modality. Results also highlight the possibility, through eye-movement analyses, to identify an ocular signature specific to the use of cognitive strategies for erectile inhibition, which is characterized by a partial avoidance of the sexual content and a general deceleration of the visual exploration process. This project exposes the benefits inherent to the presentation of computer-generated stimuli in virtual immersion and the use of eye-tracking devices for penile plethysmography assessments and supports the progressive implementation of this modality in clinical settings