Using EEG to decode subjective levels of emotional arousal during an immersive VR roller coaster ride

Emotional arousal is a key component of a user’s experience in immersive virtual reality (VR). Subjective and highly dynamic in nature, emotional arousal involves the whole body and particularly the brain. However, it has been difficult to relate subjective emotional arousal to an objective, neurophysiological marker—especially in naturalistic settings. We tested the association between continuously changing states of emotional arousal and oscillatory power in the brain during a VR roller coaster experience. We used novel spatial filtering approaches to predict self-reported emotional arousal from the electroencephalogram (EEG) signal of 38 participants. Periods of high vs. low emotional arousal could be classified with accuracies significantly above chance level. Our results are consistent with prior findings regarding emotional arousal in less naturalistic settings. We demonstrate a new approach to decode states of subjective emotional arousal from continuous EEG data in an immersive VR experience

Decoding subjective emotional arousal from EEG during an immersive Virtual Reality experience

Immersive virtual reality (VR) enables naturalistic neuroscientific studies while maintaining experimental control, but dynamic and interactive stimuli pose methodological challenges. We here probed the link between emotional arousal, a fundamental property of affective experience, and parieto-occipital alpha power under naturalistic stimulation:37 young healthy adults completed an immersive VR experience, which included rollercoaster rides, while their EEG was recorded. They then continuously rated their subjective emotional arousal while viewing a replay of their experience. The association between emotional arousal and parieto-occipital alpha power was tested and confirmed by (1) decomposing the continuous EEG signal while maximizing the comodulation between alpha power and arousal ratings and by (2) decoding periods of high and low arousal with discriminative common spatial patterns and a Long Short-Term Memory recurrent neural network.We successfully combine EEG and a naturalistic immersive VR experience to extend previous findings on the neurophysiology of emotional arousal towards real-world neuroscience.Competing Interest StatementThe authors have declared no competing interest

Decoding subjective emotional arousal from eeg during an immersive virtual reality experience

Immersive virtual reality (VR) enables naturalistic neuroscientific studies while maintaining experimental control, but dynamic and interactive stimuli pose methodological challenges. We here probed the link between emotional arousal, a fundamental property of affective experience, and parieto-occipital alpha power under naturalistic stimulation: 37 young healthy adults completed an immersive VR experience, which included rollercoaster rides, while their EEG was recorded. They then continuously rated their subjective emotional arousal while viewing a replay of their experience. The association between emotional arousal and parieto-occipital alpha power was tested and confirmed by (1) decomposing the continuous EEG signal while maximizing the comodulation between alpha power and arousal ratings and by (2) decoding periods of high and low arousal with discriminative common spatial patterns and a Long Short-Term Memory recurrent neural network. We successfully combine EEG and a naturalistic immersive VR experience to extend previous findings on the neurophysiology of emotional arousal towards real-world neuroscience

Towards EEG-Based Haptic Interaction within Virtual Environments

Current virtual environments (VE) enable perceiving haptic stimuli to facilitate 3D user interaction, but lack brain-interfacial contents. Using electroencephalography (EEG), we undertook a feasibility study on exploring event-related potential (ERP) patterns of the user's brain responses during haptic interaction within a VE. The interaction was flying a virtual drone along a curved transmission line to detect defects under the stimuli (e.g., force increase and/or vibrotactile cues). We found that there were variations in the peak amplitudes and latencies (as ERP patterns) of the responses at about 200 ms post the onset of the stimuli. The largest negative peak occurred during 200~400 ms after the onset in all vibration-related blocks. Moreover, the amplitudes and latencies of the peak were differentiable among the vibration-related blocks. These findings imply feasible decoding of the brain responses during haptic interaction within VEs

OpenVirtualObjects (OVO): An open set of standardized and validated 3D household objects for virtual reality-based research, assessment, and therapy

Virtual reality (VR) technology provides clinicians, therapists, and researchers with new opportunities to observe, assess, and train behaviour in realistic yet well-controlled environments. However, VR also comes with a number of challenges. For example, compared to more abstract experiments and tests on 2D computer screens, VR-based tasks are more complex to create, which can make it more expensive and time-consuming. One way to overcome these challenges is to create, standardize, and validate VR content and to make it openly available for researchers and clinicians. Here we introduce the OpenVirtualObjects (OVO), a set of 124 realistic 3D household objects that people encounter and use in their everyday lives. The objects were rated by 34 younger and 25 older adults for recognizability, familiarity, details (i.e., visual complexity), contact, and usage (i.e., frequency of usage in daily life). All participants also named and categorized the objects. We provide the data and the experiment- and analysis code online. With OVO, we hope to facilitate VR-based research and clinical applications. Easy and free availability of standardized and validated 3D objects can support systematic VR-based studies and the development of VR-based diagnostics and therapeutic tools

Exploring Emotions and Engagement: A Multi-componential Analysis Using Films and Virtual Reality

In the digital age, where our lives are intertwined with intelligent systems and immersive experiences, understanding how emotions are shaped and influenced is more crucial than ever. Despite the attention to discrete and dimensional models, neuroscientific evidence supports that emotions are complex and multi-faceted. While the Component Process Model (CPM) acknowledges the complexity of emotions through its five interconnected components: appraisal, motivation, physiology, expression, and feeling, it has received limited attention in Affective Computing. Despite some recent advances in full CPM research, limitations exist. The relatively narrow emphasis on full CPM has resulted in a scarcity of available datasets for in-depth exploration. Most of these datasets are film-based, with only one in Virtual Reality (VR), and all have received limited computational analysis, especially in exploratory and Machine Learning aspects. Passive film-based emotion induction has merits and limitations, as it positions participants as observers. Introducing active VR stimuli can enhance emotion elicitation due to its immersive nature, but current CPM VR analyses rely on subjective reports. VR as an empathy machine is often identified in cutting-edge emotion research; however, limited attention has been given to understanding these attributes of VR, such as engagement. This thesis aims to comprehend emotions through full CPM with computational models. It starts with analysing a film-based dataset having subjective and objective measures and presents the role of physiology in emotion discrimination. Subsequently, we underscore the significance of micro-level annotations using another film-based dataset with larger continuous subjective annotations. The thesis also introduces a data-driven approach using interactive VR games and collected multimodal measures (self-reports, physiological, facial expressions, and movements) from 39 participants. The new dataset shows the role of different components in emotion differentiation when emotions are induced actively. Furthermore, the thesis presents an innovative approach to measuring engagement in VR games. We examine the simultaneous occurrence of player motivation and physiological responses to explore potential associations with body movements. Our explorations into emotions and engagement within a multi-componential framework, utilising both films and VR games, present numerous opportunities for advancing our understanding of human behaviour and interactions to foster a more empathetic world

Modelling the use of 3D video on the quality of experience

Последњих година, очигледан је брз развој различитих медија у различитим сферама као што су потрошачка електроника, аутомобилска инфо-забава (енгл. Infotainment), софтверa у сврху здравства итд. Због тога намеће се потреба за иновативним методама процене квалитета доживљаја (енгл. Quality of Experience - QoE) које корисници доживљавају као замену за задовољство потрошача таквих системима и услугама. Емоционално стање корисника игра кључну улогу у области QoE; стога га је неопходно узети у обзир приликом процене корисничког искуства и процеса дизајнирања 3Д видео садржаја. У овој докторској дисертацији представљено је моделовање проценитеља квалитета доживљава заснованог на повратној вишеслојној вештачкој неуронској мрежи као одговарајућој техници машинског учења за процену човековог емоционалног стања током гледања различитих типова 3Д видео садржаја. Циљ је дизајнирање проценитеља емоционалног стања на основу директних психо-физиолошких мерења. Разматрани психо- физиолошки сигнали укључују срчану фреквенцију (HR) израчунату на основу ехо-кардиограма (ECG), електро-дермалну активност (EDA) и активност мозга (BA) у електро-енцефалографским (EEG) сигналима. Експериментални део истраживања постављен је тако да су учесници гледали серију 3Д видео садржаја који се разликују у погледу визуелног квалитета и типа садржаја, док су поменути психо-физиолошки сигнали забележени помоћу специјалних сонди постављених у моменту гледања садржаја, а субјективно проживљене емоције пријављене помоћу упитника за самопроцену (SAM). Добијени резултати показују да је могуће конструисати тако високо прецизан процењивач емоционалних стања.Poslednjih godina, očigledan je brz razvoj različitih medija u različitim sferama kao što su potrošačka elektronika, automobilska info-zabava (engl. Infotainment), softvera u svrhu zdravstva itd. Zbog toga nameće se potreba za inovativnim metodama procene kvaliteta doživljaja (engl. Quality of Experience - QoE) koje korisnici doživljavaju kao zamenu za zadovoljstvo potrošača takvih sistemima i uslugama. Emocionalno stanje korisnika igra ključnu ulogu u oblasti QoE; stoga ga je neophodno uzeti u obzir prilikom procene korisničkog iskustva i procesa dizajniranja 3D video sadržaja. U ovoj doktorskoj disertaciji predstavljeno je modelovanje procenitelja kvaliteta doživljava zasnovanog na povratnoj višeslojnoj veštačkoj neuronskoj mreži kao odgovarajućoj tehnici mašinskog učenja za procenu čovekovog emocionalnog stanja tokom gledanja različitih tipova 3D video sadržaja. Cilj je dizajniranje procenitelja emocionalnog stanja na osnovu direktnih psiho-fizioloških merenja. Razmatrani psiho- fiziološki signali uključuju srčanu frekvenciju (HR) izračunatu na osnovu eho-kardiograma (ECG), elektro-dermalnu aktivnost (EDA) i aktivnost mozga (BA) u elektro-encefalografskim (EEG) signalima. Eksperimentalni deo istraživanja postavljen je tako da su učesnici gledali seriju 3D video sadržaja koji se razlikuju u pogledu vizuelnog kvaliteta i tipa sadržaja, dok su pomenuti psiho-fiziološki signali zabeleženi pomoću specijalnih sondi postavljenih u momentu gledanja sadržaja, a subjektivno proživljene emocije prijavljene pomoću upitnika za samoprocenu (SAM). Dobijeni rezultati pokazuju da je moguće konstruisati tako visoko precizan procenjivač emocionalnih stanja.In recent years, the rapid development of diverse media has been evident in disparate fields such as consumer electronics, automotive infotainment and healthcare software. There is a need for innovative methods to assess user perceived Quality of Experience (QoE), as a proxy for consumer satisfaction with such systems and services. Users emotional state plays a key role in QoE; thus, it is necessary to consider it in user experience evaluation and the design process of stereoscopic 3D video content. In the PhD thesis the use of a specially designed model based on a feedforward Multilayer Perception Artificial Neural Network as an appropriate Machine Learning technique for the estimation of human emotional state while viewing various categories of stereoscopic 3D video content is introduced. The goal is to design an emotional state estimator based on direct psychophysiological measurements. The considered psychophysiological signals include heart rate (HR) calculated from an echocardiogram (ECG), electro-dermal activity (EDA), and brain activity (BA) in EEG signals. In the experimental part of study, participants watched a series of 3D video contents varying in terms of visual quality and type of content, while the mentioned psychophysiological signals were recorded via specific equipment, and self-reported subjectively experienced emotions using a Self-Assessment Manikin (SAM) questionnaire. The obtained results show that it is possible to construct such a highly precise estimator of emotional state

Enhancing our lives with immersive virtual reality

Virtual reality (VR) started about 50 years ago in a form we would recognize today [stereo head-mounted display (HMD), head tracking, computer graphics generated images] – although the hardware was completely different. In the 1980s and 1990s, VR emerged again based on a different generation of hardware (e.g., CRT displays rather than vector refresh, electromagnetic tracking instead of mechanical). This reached the attention of the public, and VR was hailed by many engineers, scientists, celebrities, and business people as the beginning of a new era, when VR would soon change the world for the better. Then, VR disappeared from public view and was rumored to be “dead.” In the intervening 25 years a huge amount of research has nevertheless been carried out across a vast range of applications – from medicine to business, from psychotherapy to industry, from sports to travel. Scientists, engineers, and people working in industry carried on with their research and applications using and exploring different forms of VR, not knowing that actually the topic had already passed away. The purpose of this article is to survey a range of VR applications where there is some evidence for, or at least debate about, its utility, mainly based on publications in peer-reviewed journals. Of course not every type of application has been covered, nor every scientific paper (about 186,000 papers in Google Scholar): in particular, in this review we have not covered applications in psychological or medical rehabilitation. The objective is that the reader becomes aware of what has been accomplished in VR, where the evidence is weaker or stronger, and what can be done. We start in Section 1 with an outline of what VR is and the major conceptual framework used to understand what happens when people experience it – the concept of “presence.” In Section 2, we review some areas where VR has been used in science – mostly psychology and neuroscience, the area of scientific visualization, and some remarks about its use in education and surgical training. In Section 3, we discuss how VR has been used in sports and exercise. In Section 4, we survey applications in social psychology and related areas – how VR has been used to throw light on some social phenomena, and how it can be used to tackle experimentally areas that cannot be studied experimentally in real life. We conclude with how it has been used in the preservation of and access to cultural heritage. In Section 5, we present the domain of moral behavior, including an example of how it might be used to train professionals such as medical doctors when confronting serious dilemmas with patients. In Section 6, we consider how VR has been and might be used in various aspects of travel, collaboration, and industry. In Section 7, we consider mainly the use of VR in news presentation and also discuss different types of VR. In the concluding Section 8, we briefly consider new ideas that have recently emerged – an impossible task since during the short time we have written this page even newer ideas have emerged! And, we conclude with some general considerations and speculations. Throughout and wherever possible we have stressed novel applications and approaches and how the real power of VR is not necessarily to produce a faithful reproduction of “reality” but rather that it offers the possibility to step outside of the normal bounds of reality and realize goals in a totally new and unexpected way. We hope that our article will provoke readers to think as paradigm changers, and advance VR to realize different worlds that might have a positive impact on the lives of millions of people worldwide, and maybe even help a little in saving the planet

Using EEG to decode subjective levels of emotional arousal during an immersive VR roller coaster ride

Emotional arousal is a key component of a user’s experience in immersive virtual reality (VR). Subjective and highly dynamic in na- ture, emotional arousal involves the whole body and particularly the brain. However, it has been difficult to relate subjective emotional arousal to an objective, neurophysiological marker—especially in naturalistic settings. We tested the association between continuously changing states of emotional arousal and oscillatory power in the brain during a VR roller coaster experience. We used novel spatial filtering approaches to predict self-reported emotional arousal from the electroencephalogram (EEG) signal of 38 participants. Periods of high vs. low emotional arousal could be classified with accuracies significantly above chance level. Our results are consistent with prior findings regarding emotional arousal in less naturalistic settings. We demonstrate a new approach to decode states of subjective emotional arousal from continuous EEG data in an immersive VR experience

Réalité évoquée, des rêves aux simulations (un cadre conceptuel de la réalité au regard de la présence)

Dans cette recherche, nous présentons le concept de Réalité Évoquée ( Evoked Reality ) afin d'essayer de relier différentes notions entourant la présence et la réalité au sein d'un cadre commun. Nous introduisons et illustrons le concept en tant que illusion de la réalité (Réalité Évoquée) qui évoque un sentiment de présence (Présence Évoquée) dans nos esprits. Nous distinguons les concepts de Réalité Média-Évoquée et Réalité Auto-Évoquée et nous les définissons clairement. Le concept de Réalité Évoquée nous permet d'introduire un modèle tripolaire de la réalité, qui remet en cause le modèle classique des deux pôles. Nous présentons également un modèle graphique appelé Reality-Presence Map (Carte Réalité-Présence) qui nous permet de localiser et d'analyser toutes les expériences cognitives concernant la présence et la réalité. Nous explorons également les qualia et la subjectivité de nos expériences de Réalité Évoquée. Deux expériences ont été réalisées : l'une dans le domaine de la Réalité Média-Évoquée et l'autre dans celui de l'Auto-Évoquée. Les expériences nous ont permis de valider nos hypothèses et de réaliser que nos recherches empiriques pouvaient encore être poussées plus loin encore. Enfin, nous illustrons les différentes implications et nous examinons les applications et les utilisations possibles de notre concept, en particulier dans le domaine de la recherche sur la présence. En outre, nous proposons d'étendre la recherche sur la présence au-delà du domaine de la réalité virtuelle et des moyens de communication et de l'étudier dans une perspective plus large que celle des sciences cognitives. Nous sommes convaincus que ce concept de Réalité Évoquée et le modèle proposé peuvent avoir des applications significatives dans l'étude de la présence et dans l'exploration des possibilités qui dépassent la réalité virtuelle.In this research, we introduce the concept of "Evoked Reality" in an attempt to bring together various ideas on presence and reality onto a common platform. The concept we propose and illustrate is in fact an 'illusion of reality' (Evoked Realty) that simply evokes a 'sense of presence' (Evoked Presence) in our minds. We clearly define and differentiate between a Media-Evoked and a Self-Evoked Reality. That helped us introduce the Three Pole Reality Model that redefines the classical Two Pole Reality Model. We also present a graphical model called Reality-Presence Map, which would help us locate and analyse every possible cognitive experience relating to presence and reality. We also explore the qualia and subjectivity of our experiences of Evoked Reality. Two experiments were conducted, one in the area of Media-Evoked Reality and one in Self-Evoked Reality. The experiments in fact lead to fruitful conclusions regarding our hypotheses and help us understand what could be further empirically studied. Ultimately, we illustrate different implications and shed light on prospective applications and uses of our concept, especially in the area of research on presence. In addition, we strongly suggest that we must open up presence research beyond the domain of virtual reality and communication media, and examine it from a broader perspective of cognitive science. We strongly believe that this concept of Evoked Reality and the proposed model may have significant applications in the study of presence, and in exploring the possibilities beyond virtual reality.PARIS-Arts et Métiers (751132303) / SudocSudocFranceF