7 research outputs found
Towards EEG-Based Haptic Interaction within Virtual Environments
Current virtual environments (VE) enable perceiving haptic stimuli to facilitate 3D user interaction, but lack brain-interfacial contents. Using electroencephalography (EEG), we undertook a feasibility study on exploring event-related potential (ERP) patterns of the user's brain responses during haptic interaction within a VE. The interaction was flying a virtual drone along a curved transmission line to detect defects under the stimuli (e.g., force increase and/or vibrotactile cues). We found that there were variations in the peak amplitudes and latencies (as ERP patterns) of the responses at about 200 ms post the onset of the stimuli. The largest negative peak occurred during 200~400 ms after the onset in all vibration-related blocks. Moreover, the amplitudes and latencies of the peak were differentiable among the vibration-related blocks. These findings imply feasible decoding of the brain responses during haptic interaction within VEs
" Do you feel in control? " : Towards Novel Approaches to Characterise, Manipulate and Measure the Sense of Agency in Virtual Environments
International audienceWhile the Sense of Agency (SoA) has so far been predominantly characterised in VR as a component of the Sense of Embodiment, other communities (e.g., in psychology or neurosciences) have investigated the SoA from a different perspective proposing complementary theories. Yet, despite the acknowledged potential benefits of catching up with these theories a gap remains. This paper first aims to contribute to fill this gap by introducing a theory according to which the SoA can be divided into two components, the feeling and the judgment of agency, and relies on three principles, namely the principles of priority, exclusivity and consistency. We argue that this theory could provide insights on the factors influencing the SoA in VR systems. Second, we propose novel approaches to manipulate the SoA in controlled VR experiments (based on these three principles) as well as to measure the SoA, and more specifically its two components based on neurophysiological markers, using ElectroEncephaloGraphy (EEG). We claim that these approaches would enable us to deepen our understanding of the SoA in VR contexts. Finally, we validate these approaches in an experiment. Our results (N=24) suggest that our approach was successful in manipulating the SoA as the modulation of each of the three principles induced significant decreases of the SoA (measured using questionnaires). In addition, we recorded participants' EEG signals during the VR experiment, and neurophysiological markers of the SoA, potentially reflecting the feeling and judgment of agency specifically, were revealed. Our results also suggest that users' profile, more precisely their Locus of Control (LoC), influences their level of immersion and SoA
Effects of voluntary heart rate control on user engagement and agency in a virtual reality game
It has been demonstrated that virtual reality (VR) exposure can afect the subjective experience of diferent situations, cognitive capabilities or behavior. It is known that there is a link between a personâs physiological state and their psychological self-report and user experience. As an immersive experience can afect usersâ physiological data, it is possible to adapt and enhance the content of a virtual environment in real-time base on physiological data feedback (biofeedback). With the rapid evolution of the physiological monitoring technologies, it is now possible to exploit diferent modalities of biofeedback, in a cheap and non-cumbersome manner, and study how they can afect user experience. While most of the studies involving physiological data use it as a measuring tool, we want to study its impact when direct and voluntary physiological control becomes a mean of interaction. To do so, we created a two-parts protocol. The frst part was designed to categorize the participants on their heart rate control competency. In the second part of the study, we immersed our participants in a VR experience where they must control their heart rate to interact with the elements in the game. The results were analyzed based on the competency distribution. We observed consistent results between our competency scale and the participantsâ control of the biofeedback game mechanic. We also found that our direct biofeedback mechanic is highly engaging. We observed that it generated a strong feeling of agency, which is linked with usersâ level of heart rate control. We highlighted the richness of biofeedback as a direct game mechanic, prompting interesting perspective for personalized immersive experiences
Ătude de lâimpact du retour haptique sur le sentiment dâincarnation
International audienceLe rapport ci-aprĂšs cherche Ă rendre compte du travail effectuĂ© sur lâimpact dâun retour haptique sur le sentiment dâincarnation en environnement virtuel immersif. Ce travail sâest effectuĂ© dans le cadre dâun stage de six mois au sein du centre Inria Lille. Nous avons dĂ©veloppĂ© une expĂ©rience contrĂŽlĂ©e avec trois conditions. Deux conditions proposent une forme de retour haptique diffĂ©rente, une condition Ă retour dâeffort et une condition Ă retour vibro-tactile, et nous les avons comparĂ©es Ă une condition de contrĂŽle, sans retour. Le pĂ©riphĂ©rique haptique utilisĂ© est un bras Ă retour dâeffort dont lâinterface tangible est un stylet. Les donnĂ©es acquises ont Ă©tĂ© obtenues au travers dâun questionnaire Ă©valuant le sentiment dâincarnation. Nos rĂ©sultats montrent que les deux formes de retour haptique mises en place suscitent un niveau dâincarnation plus Ă©levĂ©s chez les participants. Les diffĂ©rences trouvĂ©es entre les conditions de contrĂŽle et Ă retour dâeffort permettent dâaffirmer la fidĂ©litĂ© du retour. A lâopposĂ©, lâabsence de rĂ©sultats significatifs plus prĂ©cis et les retours mitigĂ©s des participants nous laissent penser que le retour vibro-tactile nâest pas assez cohĂ©rent pour susciter les mĂȘmes niveaux dâincarnation que le retour dâeffort
A Multi-Modal, Modified-Feedback and Self-Paced Brain-Computer Interface (BCI) to Control an Embodied Avatar's Gait
Brain-computer interfaces (BCI) have been used to control the gait of a virtual self-avatar with the
aim of being used in gait rehabilitation. A BCI decodes the brain signals representing a desire to
do something and transforms them into a control command for controlling external devices.
The feelings described by the participants when they control a self-avatar in an immersive virtual
environment (VE) demonstrate that humans can be embodied in the surrogate body of an avatar
(ownership illusion). It has recently been shown that inducing the ownership illusion and then
manipulating the movements of oneâs self-avatar can lead to compensatory motor control
strategies.
In order to maximize this effect, there is a need for a method that measures and monitors
embodiment levels of participants immersed in virtual reality (VR) to induce and maintain a strong
ownership illusion. This is particularly true given that reaching a high level of both BCI
performance and embodiment are inter-connected. To reach one of them, the second must be
reached as well. Some limitations of many existing systems hinder their adoption for
neurorehabilitation: 1- some use motor imagery (MI) of movements other than gait; 2- most
systems allow the user to take single steps or to walk but do not allow both, which prevents users
from progressing from steps to gait; 3- most of them function in a single BCI mode (cue-paced or
self-paced), which prevents users from progressing from machine-dependent to machine-independent
walking. Overcoming the aforementioned limitations can be done by combining
different control modes and options in one single system. However, this would have a negative
impact on BCI performance, therefore diminishing its usefulness as a potential rehabilitation tool.
In this case, there will be a need to enhance BCI performance. For such purpose, many techniques
have been used in the literature, such as providing modified feedback (whereby the presented
feedback is not consistent with the userâs MI), sequential training (recalibrating the classifier as
more data becomes available).
This thesis was developed over 3 studies. The objective in study 1 was to investigate the possibility
of measuring the level of embodiment of an immersive self-avatar, during the performing,
observing and imagining of gait, using electroencephalogram (EEG) techniques, by presenting
visual feedback that conflicts with the desired movement of embodied participants.
The objective of study 2 was to develop and validate a BCI to control single steps and forward
walking of an immersive virtual reality (VR) self-avatar, using mental imagery of these actions, in
cue-paced and self-paced modes. Different performance enhancement strategies were
implemented to increase BCI performance.
The data of these two studies were then used in study 3 to construct a generic classifier that could
eliminate offline calibration for future users and shorten training time.
Twenty different healthy participants took part in studies 1 and 2. In study 1, participants wore an
EEG cap and motion capture markers, with an avatar displayed in a head-mounted display (HMD)
from a first-person perspective (1PP). They were cued to either perform, watch or imagine a single
step forward or to initiate walking on a treadmill. For some of the trials, the avatar took a step with
the contralateral limb or stopped walking before the participant stopped (modified feedback).
In study 2, participants completed a 4-day sequential training to control the gait of an avatar in
both BCI modes. In cue-paced mode, they were cued to imagine a single step forward, using their
right or left foot, or to walk forward. In the self-paced mode, they were instructed to reach a target
using the MI of multiple steps (switch control mode) or maintaining the MI of forward walking
(continuous control mode). The avatar moved as a response to two calibrated regularized linear
discriminant analysis (RLDA) classifiers that used the Ό power spectral density (PSD) over the
foot area of the motor cortex as features. The classifiers were retrained after every session. During
the training, and for some of the trials, positive modified feedback was presented to half of the
participants, where the avatar moved correctly regardless of the participantâs real performance.
In both studies, the participantsâ subjective experience was analyzed using a questionnaire. Results
of study 1 show that subjective levels of embodiment correlate strongly with the power differences
of the event-related synchronization (ERS) within the Ό frequency band, and over the motor and
pre-motor cortices between the modified and regular feedback trials.
Results of study 2 show that all participants were able to operate the cued-paced BCI and the selfpaced
BCI in both modes. For the cue-paced BCI, the average offline performance (classification
rate) on day 1 was 67±6.1% and 86±6.1% on day 3, showing that the recalibration of the classifiers
enhanced the offline performance of the BCI (p < 0.01). The average online performance was
85.9±8.4% for the modified feedback group (77-97%) versus 75% for the non-modified feedback
group. For self-paced BCI, the average performance was 83% at switch control and 92% at
continuous control mode, with a maximum of 12 seconds of control. Modified feedback enhanced
BCI performances (p =0.001). Finally, results of study 3 show that the constructed generic models
performed as well as models obtained from participant-specific offline data. The results show that
there it is possible to design a participant-independent zero-training BCI.Les interfaces cerveau-ordinateur (ICO) ont Ă©tĂ© utilisĂ©es pour contrĂŽler la marche d'un Ă©go-avatar virtuel dans le but d'ĂȘtre utilisĂ©es dans la rĂ©adaptation de la marche. Une ICO dĂ©code les signaux du cerveau reprĂ©sentant un dĂ©sir de faire produire un mouvement et les transforme en une commande de contrĂŽle pour contrĂŽler des appareils externes.
Les sentiments dĂ©crits par les participants lorsqu'ils contrĂŽlent un Ă©go-avatar dans un environnement virtuel immersif dĂ©montrent que les humains peuvent ĂȘtre incarnĂ©s dans un corps d'un avatar (illusion de propriĂ©tĂ©). Il a Ă©tĂ© rĂ©cemment dĂ©montrĂ© que provoquer lâillusion de propriĂ©tĂ© puis manipuler les mouvements de lâĂ©go-avatar peut conduire Ă des stratĂ©gies de contrĂŽle moteur compensatoire.
Afin de maximiser cet effet, il existe un besoin d'une mĂ©thode qui mesure et surveille les niveaux dâincarnation des participants immergĂ©s dans la rĂ©alitĂ© virtuelle (RV) pour induire et maintenir une forte illusion de propriĂ©tĂ©.
D'autre part, atteindre un niveau Ă©levĂ© de performances (taux de classification) ICO et dâincarnation est interconnectĂ©. Pour atteindre l'un d'eux, le second doit Ă©galement ĂȘtre atteint. Certaines limitations de plusieurs de ces systĂšmes entravent leur adoption pour la neurorĂ©habilitation: 1- certains utilisent l'imagerie motrice (IM) des mouvements autres que la marche; 2- la plupart des systĂšmes permettent Ă l'utilisateur de faire des pas simples ou de marcher mais pas les deux, ce qui ne permet pas Ă un utilisateur de passer des pas Ă la marche; 3- la plupart fonctionnent en un seul mode dâICO, rythmĂ© (cue-paced) ou auto-rythmĂ© (self-paced). Surmonter les limitations susmentionnĂ©es peut ĂȘtre fait en combinant diffĂ©rents modes et options de commande dans un seul systĂšme. Cependant, cela aurait un impact nĂ©gatif sur les performances de lâICO, diminuant ainsi son utilitĂ© en tant qu'outil potentiel de rĂ©habilitation. Dans ce cas, il sera nĂ©cessaire d'amĂ©liorer les performances des ICO. Ă cette fin, de nombreuses techniques ont Ă©tĂ© utilisĂ©es dans la littĂ©rature, telles que la rĂ©troaction modifiĂ©e, le recalibrage du classificateur et l'utilisation d'un classificateur gĂ©nĂ©rique.
Le projet de cette thĂšse a Ă©tĂ© rĂ©alisĂ© en 3 Ă©tudes, avec objectif d'Ă©tudier dans l'Ă©tude 1, la possibilitĂ© de mesurer le niveau d'incarnation d'un Ă©go-avatar immersif, lors de l'exĂ©cution, de l'observation et de l'imagination de la marche, Ă l'aide des techniques encĂ©phalogramme (EEG), en prĂ©sentant une rĂ©troaction visuelle qui entre en conflit avec la commande du contrĂŽle moteur des sujets incarnĂ©s. L'objectif de l'Ă©tude 2 Ă©tait de dĂ©velopper un BCI pour contrĂŽler les pas et la marche vers lâavant d'un Ă©go-avatar dans la rĂ©alitĂ© virtuelle immersive, en utilisant l'imagerie motrice de ces actions, dans des modes rythmĂ©s et auto-rythmĂ©s. DiffĂ©rentes stratĂ©gies d'amĂ©lioration des performances ont Ă©tĂ© mises en Ćuvre pour augmenter la performance (taux de classification) de lâICO.
Les données de ces deux études ont ensuite été utilisées dans l'étude 3 pour construire des classificateurs génériques qui pourraient éliminer la calibration hors ligne pour les futurs utilisateurs et raccourcir le temps de formation.
Vingt participants sains diffĂ©rents ont participĂ© aux Ă©tudes 1 et 2. Dans l'Ă©tude 1, les participants portaient un casque EEG et des marqueurs de capture de mouvement, avec un avatar affichĂ© dans un casque de RV du point de vue de la premiĂšre personne (1PP). Ils ont Ă©tĂ© invitĂ©s Ă performer, Ă regarder ou Ă imaginer un seul pas en avant ou la marche vers lâavant (pour quelques secondes) sur le tapis roulant. Pour certains essais, l'avatar a fait un pas avec le membre controlatĂ©ral ou a arrĂȘtĂ© de marcher avant que le participant ne s'arrĂȘte (rĂ©troaction modifiĂ©e).
Dans l'Ă©tude 2, les participants ont participĂ© Ă un entrainement sĂ©quentiel de 4 jours pour contrĂŽler la marche d'un avatar dans les deux modes de lâICO. En mode rythmĂ©, ils ont imaginĂ© un seul pas en avant, en utilisant leur pied droit ou gauche, ou la marche vers lâavant . En mode auto-rythmĂ©, il leur a Ă©tĂ© demandĂ© d'atteindre une cible en utilisant l'imagerie motrice (IM) de plusieurs pas (mode de contrĂŽle intermittent) ou en maintenir l'IM de marche vers lâavant (mode de contrĂŽle continu). L'avatar s'est dĂ©placĂ© en rĂ©ponse Ă deux classificateurs âRegularized Linear Discriminant Analysisâ (RLDA) calibrĂ©s qui utilisaient comme caractĂ©ristiques la densitĂ© spectrale de puissance (Power Spectral Density; PSD) des bandes de frĂ©quences ” (8-12 Hz) sur la zone du pied du cortex moteur. Les classificateurs ont Ă©tĂ© recalibrĂ©s aprĂšs chaque session. Au cours de lâentrainement et pour certains des essais, une rĂ©troaction modifiĂ©e positive a Ă©tĂ© prĂ©sentĂ©e Ă la moitiĂ© des participants, oĂč l'avatar s'est dĂ©placĂ© correctement quelle que soit la performance rĂ©elle du participant. Dans les deux Ă©tudes, l'expĂ©rience subjective des participants a Ă©tĂ© analysĂ©e Ă l'aide d'un questionnaire.
Les rĂ©sultats de l'Ă©tude 1 montrent que les niveaux subjectifs dâincarnation sont fortement corrĂ©lĂ©s Ă la diffĂ©rence de la puissance de la synchronisation liĂ©e Ă lâĂ©vĂ©nement (Event-Related Synchronization; ERS) sur la bande de frĂ©quence ÎŒ et sur le cortex moteur et prĂ©moteur entre les essais de rĂ©troaction modifiĂ©s et rĂ©guliers. L'Ă©tude 2 a montrĂ© que tous les participants Ă©taient capables dâutiliser le BCI rythmĂ© et auto-rythmĂ© dans les deux modes. Pour le BCI rythmĂ©, la performance hors ligne moyenne au jour 1 Ă©tait de 67±6,1% et 86±6,1% au jour 3, ce qui montre que le recalibrage des classificateurs a amĂ©liorĂ© la performance hors ligne du BCI (p <0,01). La performance en ligne moyenne Ă©tait de 85,9±8,4% pour le groupe de rĂ©troaction modifiĂ© (77-97%) contre 75% pour le groupe de rĂ©troaction non modifiĂ©. Pour le BCI auto-rythmĂ©, la performance moyenne Ă©tait de 83% en commande de commutateur et de 92% en mode de commande continue, avec un maximum de 12 secondes de commande. Les performances de lâICO ont Ă©tĂ© amĂ©liorĂ©es par la rĂ©troaction modifiĂ©e (p = 0,001). Enfin, les rĂ©sultats de l'Ă©tude 3 montrent que pour la classification des initialisations des pas et de la marche, il a Ă©tĂ© possible de construire des modĂšles gĂ©nĂ©riques Ă partir de donnĂ©es hors ligne spĂ©cifiques aux participants. Les rĂ©sultats montrent la possibilitĂ© de concevoir une ICO ne nĂ©cessitant aucun entraĂźnement spĂ©cifique au participant
Share Your Reality: The effects of haptic feedback on virtual avatar co-embodiment
The advent of Virtual Reality (VR) technologies has begun a shift in communication between
people and their interaction with 3D virtual environments. VR has great potential to provide
high immersion to users, allowing designers to create vivid and impossible interactions.
However, while software and technology play a crucial role in creating a VR experience, as
designers we must understand how humans perceive these elements of sensory illusions in
order to create experiences that are appropriately received and interpreted.
Recent efforts in âSocial Virtual Realityâ explore shared experiences and collaboration
between users through remote interactions in virtual environments. One emerging concept
is âVirtual Co-embodimentâ, enabling two users to share a virtual character. This interaction
fosters a unique multiplayer experience, promoting social co-ordination and collaborative
user experiences. Co-embodiment achieves heightened levels of co-presence while still
preserving a strong sense of agency and body ownership for both the users. The influence
of feedback mechanisms on these factors is an important point of interest.
This project expands on this idea of co-embodiment by investigating how haptic feedback
affects these factors between dyads performing shared perceptual activities. To examine
these effects, an experiment was designed wherein pairs of participants in co-embodiment,
performed reaching tasks with varying levels of control over the shared hand avatar, both
with and without haptic feedback conditions. This was facilitated using a VR system that
was tailor-made to meet these requirements. Objective measurements of their motion were
collected during the interaction and subjective responses were recorded post-interaction.
The results showed that participants sense of agency was significantly lower in conditions
where they received haptic feedback when their hand positions overlapped, compared to
conditions where there was no haptic feedback. Participants made negative associations of
the haptic feedback during the experiment as expressed in the post-experiment interviews,
which could have affected their perceptions of agency. They also show significantly greater
sense of agency during tasks where they shared a common target with their partner, while
co-presence and embodiment levels were significantly higher in tasks where there were
multiple targets. Participants also spontaneously adopted leader and follower roles during
the interactions with different motion strategies to gain control over the shared avatar.
These, along with other findings of the qualitative and quantitative analysis are compiled
to extract insights to inform future research of this concept. Additionally, limitations of the
study are discussed along with recommendations for further improvements to enhance this
paradigm