Virtual reality obstacle crossing: adaptation, retention and transfer to the physical world

Virtual reality (VR) paradigms are increasingly being used in movement and exercise sciences with the aim to enhance motor function and stimulate motor adaptation in healthy and pathological conditions. Locomotor training based in VR may be promising for motor skill learning, with transfer of VR skills to the physical world in turn required to benefit functional activities of daily life. This PhD project aims to examine locomotor adaptations to repeated VR obstacle crossing in healthy young adults as well as transfers to the untrained limb and the physical world, and retention potential of the learned skills. For these reasons, the current thesis comprises three studies using controlled VR obstacle crossing interventions during treadmill walking. In the first and second studies we investigated adaptation to crossing unexpectedly appearing virtual obstacles, with and without feedback about crossing performance, and its transfer to the untrained leg. In the third study we investigated transfer of virtual obstacle crossing to physical obstacles of similar size to the virtual ones, that appeared at the same time point within the gait cycle. We also investigated whether the learned skills can be retained in each of the environments over one week. In all studies participants were asked to walk on a treadmill while wearing a VR headset that represented their body as an avatar via real-time synchronised optical motion capture. Participants had to cross virtual and/or physical obstacles with and without feedback about their crossing performance. If applicable, feedback was provided based on motion capture immediately after virtual obstacle crossing. Toe clearance, margin of stability, and lower extremity joint angles in the sagittal plane were calculated for the crossing legs to analyse adaptation, transfer, and retention of obstacle crossing performance. The main outcomes of the first and second studies were that crossing multiple virtual obstacles increased participants’ dynamic stability and led to a nonlinear adaptation of toe clearance that was enhanced by visual feedback about crossing performance. However, independent of the use of feedback, no transfer to the untrained leg was detected. Moreover, despite significant and rapid adaptive changes in locomotor kinematics with repeated VR obstacle crossing, results of the third study revealed limited transfer of learned skills from virtual to physical obstacles. Lastly, despite full retention over one week in the virtual environment we found only partial retention when crossing a physical obstacle while walking on the treadmill. In summary, the findings of this PhD project confirmed that repeated VR obstacle perturbations can effectively stimulate locomotor skill adaptations. However, these are not transferable to the untrained limb irrespective of enhanced awareness and feedback. Moreover, the current data provide evidence that, despite significant adaptive changes in locomotion kinematics with repeated practice of obstacle crossing under VR conditions, transfer to and retention in the physical environment is limited. It may be that perception-action coupling in the virtual environment, and thus sensorimotor coordination, differs from the physical world, potentially inhibiting retained transfer between those two conditions. Accordingly, VR-based locomotor skill training paradigms need to be considered carefully if they are to replace training in the physical world

Effects of Sensorimotor Perturbations on Balance Performance and Electrocortical Dynamics

Humans must frequently adapt their posture to prevent loss of balance. Such balance control requires complex, precisely-timed coordination among sensory input, neural processing, and motor output. Despite its importance, our current understanding of cortical involvement during balance control remains limited by traditional neuroimaging methods, which are stationary and have poor time resolution. High-density electroencephalography (EEG), combined with independent component analysis, has become a promising tool for recording cortical dynamics during balance perturbations due to its portability and high temporal resolution. Additionally, recent improvements in immersive virtual reality headsets may provide new rehabilitative paradigms, but the effects of virtual reality on balance and cortical function remain poorly understood. In my first study, I recorded high-density EEG from healthy, young adult subjects as they walked along a beam with and without virtual reality high heights exposure. While virtual high heights did induce stress, the use of virtual reality during the task increased performance errors and EEG measures of cognitive loading compared to real-world viewing without a headset. In my second study, I collected high-density EEG from healthy young adults as they walked along a treadmill-mounted balance beam to determine the effect of a transient visual perturbation on training in virtual reality. Subjects in the perturbations group improved comparably to those that trained without virtual reality, indicating that the perturbation helped subjects overcome the negative effects of virtual reality on motor learning. The perturbation primarily elicited a cognitive change. In my third study, healthy, young adult EEG was recorded during physical pull and visual rotation perturbations to tandem walking and tandem standing. I found similar electrocortical patterns for both perturbation types, but different cortical areas were involved for each. In my fourth study, I used a phantom head to validate EEG connectivity methods based on Granger causality in a real-world environment. In general, connectivity measures could determine the underlying connections, but many were susceptible to high-frequency false positives. Using data from my third study, my fifth study analyzed corticomuscular connectivity patterns following sensorimotor balance perturbations. I found strong occipito-parietal connections regardless of perturbation type, along with evidence of direct muscular control from the supplementary motor area during the standing perturbation response. Taken together, the work presented in this dissertation greatly expands upon the current knowledge of cortical processing during sensorimotor balance perturbations and the effect of such perturbations on short-term motor learning, providing multiple avenues for future exploration.PHDBiomedical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/147615/1/stepeter_1.pd

Contributions of Training Programs Supported by VR Techniques to the Prevention of STF Accidents

© 2020, Springer Nature Switzerland AG. Occupational safety and health (OSH) is active at all levels of the hierarchy of controls to prevent accidents associated with slips, trips and falls (STF). Training programs related to STF prevention are increasingly supported by virtual reality (VR) techniques. A review revealed a wide range of applications in practical and scientific areas. Trainings for operational practice vary regarding objectives, target groups, application contexts, media, and effectiveness, if available. Trainings in scientific studies are well designed for specific purposes at hand, but not suitable for direct application in operational practice. Research is required to bridge the gap. An investigation on gait stability and control in a VR-based obstacle avoidance training scenario has been conducted to contribute to developments in STF prevention. Initial results indicated a high level of presence and no evidence for detrimental effects on body and gait stability through application of VR techniques. This provides a sound basis for analysis of other data still required and for guiding similar and subsequent studies along knowledge gained by training programs available

Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing

[EN] Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which allows researchers to simulate environments in controlled laboratory conditions with high levels of sense of presence and interactivity, is becoming more popular in emotion research. Moreover, its synergy with implicit measurements and machine-learning techniques has the potential to impact transversely in many research areas, opening new opportunities for the scientific community. This paper presents a systematic review of the emotion recognition research undertaken with physiological and behavioural measures using head-mounted displays as elicitation devices. The results highlight the evolution of the field, give a clear perspective using aggregated analysis, reveal the current open issues and provide guidelines for future research.This research was funded by European Commission, grant number H2020-825585 HELIOS.Marín-Morales, J.; Llinares Millán, MDC.; Guixeres Provinciale, J.; Alcañiz Raya, ML. (2020). Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing. Sensors. 20(18):1-26. https://doi.org/10.3390/s20185163S126201

A Multi-Modal, Modified-Feedback and Self-Paced Brain-Computer Interface (BCI) to Control an Embodied Avatar's Gait

Brain-computer interfaces (BCI) have been used to control the gait of a virtual self-avatar with the aim of being used in gait rehabilitation. A BCI decodes the brain signals representing a desire to do something and transforms them into a control command for controlling external devices. The feelings described by the participants when they control a self-avatar in an immersive virtual environment (VE) demonstrate that humans can be embodied in the surrogate body of an avatar (ownership illusion). It has recently been shown that inducing the ownership illusion and then manipulating the movements of one’s self-avatar can lead to compensatory motor control strategies. In order to maximize this effect, there is a need for a method that measures and monitors embodiment levels of participants immersed in virtual reality (VR) to induce and maintain a strong ownership illusion. This is particularly true given that reaching a high level of both BCI performance and embodiment are inter-connected. To reach one of them, the second must be reached as well. Some limitations of many existing systems hinder their adoption for neurorehabilitation: 1- some use motor imagery (MI) of movements other than gait; 2- most systems allow the user to take single steps or to walk but do not allow both, which prevents users from progressing from steps to gait; 3- most of them function in a single BCI mode (cue-paced or self-paced), which prevents users from progressing from machine-dependent to machine-independent walking. Overcoming the aforementioned limitations can be done by combining different control modes and options in one single system. However, this would have a negative impact on BCI performance, therefore diminishing its usefulness as a potential rehabilitation tool. In this case, there will be a need to enhance BCI performance. For such purpose, many techniques have been used in the literature, such as providing modified feedback (whereby the presented feedback is not consistent with the user’s MI), sequential training (recalibrating the classifier as more data becomes available). This thesis was developed over 3 studies. The objective in study 1 was to investigate the possibility of measuring the level of embodiment of an immersive self-avatar, during the performing, observing and imagining of gait, using electroencephalogram (EEG) techniques, by presenting visual feedback that conflicts with the desired movement of embodied participants. The objective of study 2 was to develop and validate a BCI to control single steps and forward walking of an immersive virtual reality (VR) self-avatar, using mental imagery of these actions, in cue-paced and self-paced modes. Different performance enhancement strategies were implemented to increase BCI performance. The data of these two studies were then used in study 3 to construct a generic classifier that could eliminate offline calibration for future users and shorten training time. Twenty different healthy participants took part in studies 1 and 2. In study 1, participants wore an EEG cap and motion capture markers, with an avatar displayed in a head-mounted display (HMD) from a first-person perspective (1PP). They were cued to either perform, watch or imagine a single step forward or to initiate walking on a treadmill. For some of the trials, the avatar took a step with the contralateral limb or stopped walking before the participant stopped (modified feedback). In study 2, participants completed a 4-day sequential training to control the gait of an avatar in both BCI modes. In cue-paced mode, they were cued to imagine a single step forward, using their right or left foot, or to walk forward. In the self-paced mode, they were instructed to reach a target using the MI of multiple steps (switch control mode) or maintaining the MI of forward walking (continuous control mode). The avatar moved as a response to two calibrated regularized linear discriminant analysis (RLDA) classifiers that used the μ power spectral density (PSD) over the foot area of the motor cortex as features. The classifiers were retrained after every session. During the training, and for some of the trials, positive modified feedback was presented to half of the participants, where the avatar moved correctly regardless of the participant’s real performance. In both studies, the participants’ subjective experience was analyzed using a questionnaire. Results of study 1 show that subjective levels of embodiment correlate strongly with the power differences of the event-related synchronization (ERS) within the μ frequency band, and over the motor and pre-motor cortices between the modified and regular feedback trials. Results of study 2 show that all participants were able to operate the cued-paced BCI and the selfpaced BCI in both modes. For the cue-paced BCI, the average offline performance (classification rate) on day 1 was 67±6.1% and 86±6.1% on day 3, showing that the recalibration of the classifiers enhanced the offline performance of the BCI (p < 0.01). The average online performance was 85.9±8.4% for the modified feedback group (77-97%) versus 75% for the non-modified feedback group. For self-paced BCI, the average performance was 83% at switch control and 92% at continuous control mode, with a maximum of 12 seconds of control. Modified feedback enhanced BCI performances (p =0.001). Finally, results of study 3 show that the constructed generic models performed as well as models obtained from participant-specific offline data. The results show that there it is possible to design a participant-independent zero-training BCI.Les interfaces cerveau-ordinateur (ICO) ont été utilisées pour contrôler la marche d'un égo-avatar virtuel dans le but d'être utilisées dans la réadaptation de la marche. Une ICO décode les signaux du cerveau représentant un désir de faire produire un mouvement et les transforme en une commande de contrôle pour contrôler des appareils externes. Les sentiments décrits par les participants lorsqu'ils contrôlent un égo-avatar dans un environnement virtuel immersif démontrent que les humains peuvent être incarnés dans un corps d'un avatar (illusion de propriété). Il a été récemment démontré que provoquer l’illusion de propriété puis manipuler les mouvements de l’égo-avatar peut conduire à des stratégies de contrôle moteur compensatoire. Afin de maximiser cet effet, il existe un besoin d'une méthode qui mesure et surveille les niveaux d’incarnation des participants immergés dans la réalité virtuelle (RV) pour induire et maintenir une forte illusion de propriété. D'autre part, atteindre un niveau élevé de performances (taux de classification) ICO et d’incarnation est interconnecté. Pour atteindre l'un d'eux, le second doit également être atteint. Certaines limitations de plusieurs de ces systèmes entravent leur adoption pour la neuroréhabilitation: 1- certains utilisent l'imagerie motrice (IM) des mouvements autres que la marche; 2- la plupart des systèmes permettent à l'utilisateur de faire des pas simples ou de marcher mais pas les deux, ce qui ne permet pas à un utilisateur de passer des pas à la marche; 3- la plupart fonctionnent en un seul mode d’ICO, rythmé (cue-paced) ou auto-rythmé (self-paced). Surmonter les limitations susmentionnées peut être fait en combinant différents modes et options de commande dans un seul système. Cependant, cela aurait un impact négatif sur les performances de l’ICO, diminuant ainsi son utilité en tant qu'outil potentiel de réhabilitation. Dans ce cas, il sera nécessaire d'améliorer les performances des ICO. À cette fin, de nombreuses techniques ont été utilisées dans la littérature, telles que la rétroaction modifiée, le recalibrage du classificateur et l'utilisation d'un classificateur générique. Le projet de cette thèse a été réalisé en 3 études, avec objectif d'étudier dans l'étude 1, la possibilité de mesurer le niveau d'incarnation d'un égo-avatar immersif, lors de l'exécution, de l'observation et de l'imagination de la marche, à l'aide des techniques encéphalogramme (EEG), en présentant une rétroaction visuelle qui entre en conflit avec la commande du contrôle moteur des sujets incarnés. L'objectif de l'étude 2 était de développer un BCI pour contrôler les pas et la marche vers l’avant d'un égo-avatar dans la réalité virtuelle immersive, en utilisant l'imagerie motrice de ces actions, dans des modes rythmés et auto-rythmés. Différentes stratégies d'amélioration des performances ont été mises en œuvre pour augmenter la performance (taux de classification) de l’ICO. Les données de ces deux études ont ensuite été utilisées dans l'étude 3 pour construire des classificateurs génériques qui pourraient éliminer la calibration hors ligne pour les futurs utilisateurs et raccourcir le temps de formation. Vingt participants sains différents ont participé aux études 1 et 2. Dans l'étude 1, les participants portaient un casque EEG et des marqueurs de capture de mouvement, avec un avatar affiché dans un casque de RV du point de vue de la première personne (1PP). Ils ont été invités à performer, à regarder ou à imaginer un seul pas en avant ou la marche vers l’avant (pour quelques secondes) sur le tapis roulant. Pour certains essais, l'avatar a fait un pas avec le membre controlatéral ou a arrêté de marcher avant que le participant ne s'arrête (rétroaction modifiée). Dans l'étude 2, les participants ont participé à un entrainement séquentiel de 4 jours pour contrôler la marche d'un avatar dans les deux modes de l’ICO. En mode rythmé, ils ont imaginé un seul pas en avant, en utilisant leur pied droit ou gauche, ou la marche vers l’avant . En mode auto-rythmé, il leur a été demandé d'atteindre une cible en utilisant l'imagerie motrice (IM) de plusieurs pas (mode de contrôle intermittent) ou en maintenir l'IM de marche vers l’avant (mode de contrôle continu). L'avatar s'est déplacé en réponse à deux classificateurs ‘Regularized Linear Discriminant Analysis’ (RLDA) calibrés qui utilisaient comme caractéristiques la densité spectrale de puissance (Power Spectral Density; PSD) des bandes de fréquences µ (8-12 Hz) sur la zone du pied du cortex moteur. Les classificateurs ont été recalibrés après chaque session. Au cours de l’entrainement et pour certains des essais, une rétroaction modifiée positive a été présentée à la moitié des participants, où l'avatar s'est déplacé correctement quelle que soit la performance réelle du participant. Dans les deux études, l'expérience subjective des participants a été analysée à l'aide d'un questionnaire. Les résultats de l'étude 1 montrent que les niveaux subjectifs d’incarnation sont fortement corrélés à la différence de la puissance de la synchronisation liée à l’événement (Event-Related Synchronization; ERS) sur la bande de fréquence μ et sur le cortex moteur et prémoteur entre les essais de rétroaction modifiés et réguliers. L'étude 2 a montré que tous les participants étaient capables d’utiliser le BCI rythmé et auto-rythmé dans les deux modes. Pour le BCI rythmé, la performance hors ligne moyenne au jour 1 était de 67±6,1% et 86±6,1% au jour 3, ce qui montre que le recalibrage des classificateurs a amélioré la performance hors ligne du BCI (p <0,01). La performance en ligne moyenne était de 85,9±8,4% pour le groupe de rétroaction modifié (77-97%) contre 75% pour le groupe de rétroaction non modifié. Pour le BCI auto-rythmé, la performance moyenne était de 83% en commande de commutateur et de 92% en mode de commande continue, avec un maximum de 12 secondes de commande. Les performances de l’ICO ont été améliorées par la rétroaction modifiée (p = 0,001). Enfin, les résultats de l'étude 3 montrent que pour la classification des initialisations des pas et de la marche, il a été possible de construire des modèles génériques à partir de données hors ligne spécifiques aux participants. Les résultats montrent la possibilité de concevoir une ICO ne nécessitant aucun entraînement spécifique au participant

Sensorimotor control of gait: a novel approach for the study of the interplay of visual and proprioceptive feedback

Sensorimotor control theories propose that the central nervous system exploits expected sensory consequences generated by motor commands for movement planning, as well as online sensory feedback for comparison with expected sensory feedback for monitoring and correcting, if needed, ongoing motor output. In our study, we tested this theoretical framework by quantifying the functional role of expected versus actual proprioceptive feedback for planning and regulation of gait in humans. We addressed this question by using a novel methodological approach to deliver fast perturbations of the walking surface stiffness, in conjunction with a virtual reality system that provided visual feedback of upcoming changes of surface stiffness. In the predictable experimental condition, we asked subjects to learn associating visual feedback of changes in floor stiffness (sand patch) during locomotion to quantify kinematic and kinetic changes in gait. In the unpredictable experimental condition, we perturbed floor stiffness at unpredictable instances during the gait to characterize the gait-phase dependent strategies in recovering the locomotor cycle. For the unpredictable conditions, visual feedback of changes in floor stiffness was absent or inconsistent with tactile and proprioceptive feedback. The investigation of these perturbation-induced effects on legs kinematics revealed that visual feedback of upcoming changes in floor stiffness allows for both early (preparatory) and late (post-perturbation) changes in leg kinematics. However, when proprioceptive feedback is not available, the early responses do not occur while the late responses are preserved although in a, slightly attenuated form. The methods proposed and the preliminary results of this study open new directions for the investigation of the relative role of visual, tactile, and proprioceptive feedback on gait control, with potential implications for designing novel robot-assisted gait rehabilitation approaches

The development of a novel pitch-side concussion balance assessment: a comparison between a virtual reality based balance tool and the modified balance error scoring system

Background: Balance deficits are a key measurable marker of concussion injuries. An objective pitch-side concussion balance assessment needs to replace current subjective, insensitive, unportable tests. A novel pitch-side dual-task VR test has been developed to evoke perturbations, and measure COP path length changes, via a WBB. Aims: To establish whether a VR WBB system is effectively able to assess postural stability, by evoking perturbations, and to measure subsequent changes in COP path length. To establish whether mBESS error scores, or objective mBESS COP path lengths correlate with changes in COP path length post-perturbation. Methods: 14 female University of Birmingham hockey players aged 18-21 performed both the mBESS and the VR WBB assessment at the pitch-side. Results: The mean COP path length post-perturbation was significantly greater than pre- perturbation, as the tilt induced a compensatory sway response. SL error scores significantly correlated with SL COP path length, and COP path length percentage change from pre to post-perturbation. Conclusion: The dual-task VR WBB system effectively assesses postural stability by measuring subsequent changes in COP path length. The objective nature and plethora of information provided by the VR WBB system, heightens its appeal over the mBESS, as an assessment of postural stability

Contribution of Modified Visual Gain to Human Balance Control During Quiet, Upright Stance

Visual feedback provides critical information to support postural stability. Previous work has shown that magnifying visual feedback indirectly can improve postural control, such as by providing individuals with biofeedback during balance tasks. When studies have manipulated vision directly, the conditions have been restricted to include an absence of visual feedback and sway referenced paradigms. Therefore, this thesis aimed to understand how the gain of optic flow contributes to balance control during quiet, upright stance among healthy adults. Optic flow was amplified or reduced relative to head motion while participants stood quietly on either a firm or foam surface. Overall, when there was an increased reliance placed on the visual system by standing on foam, a tighter regulation of upright stance was observed as optic flow gain increased. Further, this thesis provided evidence that visual contributions to balance control may extend to higher frequencies of postural sway than previously theorized

Contribution of Modified Visual Gain to Human Balance Control During Quiet, Upright Stance

Visual feedback provides critical information to support postural stability. Previous work has shown that magnifying visual feedback indirectly can improve postural control, such as by providing individuals with biofeedback during balance tasks. When studies have manipulated vision directly, the conditions have been restricted to include an absence of visual feedback and sway referenced paradigms. Therefore, this thesis aimed to understand how the gain of optic flow contributes to balance control during quiet, upright stance among healthy adults. Optic flow was amplified or reduced relative to head motion while participants stood quietly on either a firm or foam surface. Overall, when there was an increased reliance placed on the visual system by standing on foam, a tighter regulation of upright stance was observed as optic flow gain increased. Further, this thesis provided evidence that visual contributions to balance control may extend to higher frequencies of postural sway than previously theorized

The Effects of Anxiety, Ageing, and Neurodegenerative Disease on Visual Reweighting for Postural Control

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonFor most people, the complex mechanisms of sensory reweighting and maintaining balance are effective in maintaining relative automaticity and stability. However, this optimal function can be jeopardized by a range of factors. Ageing, anxiety, changes in attention, and neurodegenerative disease are some of the factors that potentially affect sensory integration for balance. These factors are explored in this thesis over several empirical studies using visual perturbations delivered using virtual reality. First, I examined the effects of ageing and anxiety on the postural response to the erroneous visual experience of self-movement (Study 1). Secondly, I explored these effects in the context of Parkinson’s Disease with Freezing of Gait: a population with impaired non-visual sensory processing and increased anxiety related to motor control (Study 2). The final study attempts to isolate the effects of increased conscious control of movement on visual reweighting to examine the degree to which they account for the results of the previous studies (Study 3). Study 1 found that anxiety appears to relate to increased reliance on vision for balance, but it only partially accounts for greater reliance on vision in older adults. Study 2 found that people with PD+FOG take longer responding to the visual perturbation than healthy age-matched adults, and that freezing severity correlates with longer latency and possibly lower magnitude of response. Finally, Study 3 found no significant effect of increased conscious control of movement on reliance on vision for postural control. Overall, while anxiety does have the potential to increase reliance on visual input, increased visual dependency observed in older adults and people with PD is likely to have different origins, potentially related to physiological changes, rather than increased conscious control of movement. The General Discussion reviews the contribution of these findings to research on sensory integration and postural control, and discusses their implications, and directions for future research