The effect of visual, spatial and temporal manipulations on embodiment and action

The feeling of owning and controlling the body relies on the integration and interpretation of sensory input from multiple sources with respect to existing representations of the bodily self. Illusion paradigms involving multisensory manipulations have demonstrated that while the senses of ownership and agency are strongly related, these two components of bodily experience may be dissociable and differentially affected by alterations to sensory input. Importantly, however, much of the current literature has focused on the application of sensory manipulations to external objects or virtual representations of the self that are visually incongruent with the viewer’s own body and which are not part of the existing body representation. The current experiment used MIRAGE-mediated reality to investigate how manipulating the visual, spatial and temporal properties of the participant’s own hand (as opposed to a fake/virtual limb) affected embodiment and action. Participants viewed two representations of their right hand inside a MIRAGE multisensory illusions box with opposing visual (normal or grossly distorted), temporal (synchronous or asynchronous) and spatial (precise real location or false location) manipulations applied to each hand. Subjective experiences of ownership and agency towards each hand were measured alongside an objective measure of perceived hand location using a pointing task. The subjective sense of agency was always anchored to the synchronous hand, regardless of physical appearance and location. Subjective ownership also moved with the synchronous hand, except when both the location and appearance of the synchronous limb were incongruent with that of the real limb. Objective pointing measures displayed a similar pattern, however movement synchrony was not sufficient to drive a complete shift in perceived hand location, indicating a greater reliance on the spatial location of the real hand. The results suggest that while the congruence of self-generated movement is a sufficient driver for the sense of agency, the sense of ownership is additionally sensitive to cues about the visual appearance and spatial location of one’s own body

Embodiment Sensitivity to Movement Distortion and Perspective Taking in Virtual Reality

Despite recent technological improvements of immersive technologies, Virtual Reality suffers from severe intrinsic limitations, in particular the immateriality of the visible 3D environment. Typically, any simulation and manipulation in a cluttered environment would ideally require providing feedback of collisions to every body parts (arms, legs, trunk, etc.) and not only to the hands as has been originally explored with haptic feedback. This thesis addresses these limitations by relying on a cross modal perception and cognitive approach instead of haptic or force feedback. We base our design on scientific knowledge of bodily self-consciousness and embodiment. It is known that the instantaneous experience of embodiment emerges from the coherent multisensory integration of bodily signals taking place in the brain, and that altering this mechanism can temporarily change how one perceives properties of their own body. This mechanism is at stake during a VR simulation, and this thesis explores the new venues of interaction design based on these fundamental scientific findings about the embodied self. In particular, we explore the use of third person perspective (3PP) instead of permanently offering the traditional first person perspective (1PP), and we manipulate the user-avatar motor mapping to achieve a broader range of interactions while maintaining embodiment. We are guided by two principles, to explore the extent to which we can enhance VR interaction through the manipulation of bodily aspects, and to identify the extent to which a given manipulation affects the embodiment of a virtual body. Our results provide new evidence supporting strong embodiment of a virtual body even when viewed from 3PP, and in particular that voluntarily alternating point of view between 1PP and 3PP is not detrimental to the experience of ownership over the virtual body. Moreover, detailed analysis of movement quality show highly similar reaching behavior in both perspective conditions, and only obvious advantages or disadvantages of each perspective depending on the situation (e.g. occlusion of target by the body in 3PP, limited field of view in 1PP). We also show that subjects are insensitive to visuo-proprioceptive movement distortions when the nature of the distortion was not made explicit, and that subjects are biased toward self-attributing distorted movements that make the task easier

Bodily resonance: Exploring the effects of virtual embodiment on pain modulation and the fostering of empathy toward pain sufferers

Globally, around 20% of people suffer from chronic pain, an illness that cannot be cured and has been linked to numerous physical and mental conditions. According to the BioPsychoSocial model of pain, chronic pain presents patients with biological, psychological, and social challenges and difficulties. Immersive virtual reality (VR) has shown great promise in helping people manage acute and chronic pain, and facilitating empathy of vulnerable populations. Therefore, the first research trajectory of this dissertation targets chronic pain patients’ biological and psychological sufferings to provide VR analgesia, and the second research trajectory targets healthy people to build empathy and reduce patients’ social stigma. Researchers have taken the attention distraction approach to study how acute pain patients can manage their condition in VR, while the virtual embodiment approach has mostly been studied with healthy people exposed to pain stimulus. My first research trajectory aimed to understand how embodied characteristics affect users’ sense of embodiment and pain. Three studies have been carried out with healthy people under heat pain, complex regional pain syndrome patients, and phantom limb pain patients. My findings indicate that for all three studies, when users see a healthy or intact virtual body or body parts, they experience significant reductions in their self-reported pain ratings. Additionally, I found that the appearance of a virtual body has a significant impact on pain, whereas the virtual body’s motions do not. Despite the prevalence of chronic pain, public awareness of it is remarkably low, and pain patients commonly experience social stigma. Thus, having an embodied perspective of chronic pain patients is critical to understand their social stigma. Although there is a growing interest in using embodied VR to foster empathy towards gender or racial bias, few studies have focused on people with chronic pain. My second trajectory explored how researchers can foster empathy towards pain patients in embodied VR. To conclude, this dissertation uncovers the role of VR embodiment and dissects embodied characteristics in pain modulation and empathy generation. Finally, I summarized a novel conceptual design framework for embodied VR applications with design recommendations and future research directions

Virtual Body Ownership Illusions for Mental Health: A Narrative Review.

Over the last 20 years, virtual reality (VR) has been widely used to promote mental health in populations presenting different clinical conditions. Mental health does not refer only to the absence of psychiatric disorders but to the absence of a wide range of clinical conditions that influence people\u2019s general and social well-being such as chronic pain, neurological disorders that lead to motor o perceptual impairments, psychological disorders that alter behaviour and social cognition, or physical conditions like eating disorders or present in amputees. It is known that an accurate perception of oneself and of the surrounding environment are both key elements to enjoy mental health and well-being, and that both can be distorted in patients suffering from the clinical conditions mentioned above. In the past few years, multiple studies have shown the effectiveness of VR to modulate such perceptual distortions of oneself and of the surrounding environment through virtual body ownership illusions. This narrative review aims to review clinical studies that have explored the manipulation of embodied virtual bodies in VR for improving mental health, and to discuss the current state of the art and the challenges for future research in the context of clinical care

Using virtual objects with hand-tracking: the effects of visual congruence and mid-air haptics on sense of agency

Virtual reality expands the possibilities of human action. With hand-tracking technology, we can directly interact with these environments without the need for a mediating controller. Much previous research has looked at the user-avatar relationship. Here we explore the avatar-object relationship by manipulating the visual congruence and haptic feedback of the virtual object of interaction. We examine the effect of these variables on the sense of agency (SoA), which refers to the feeling of control over our actions and their effects. This psychological variable is highly relevant to user experience and is attracting increased interest in the field. Our results showed that implicit SoA was not significantly affected by visual congruence and haptics. However, both of these manipulations significantly affected explicit SoA, which was strengthened by the presence of mid-air haptics and was weakened by the presence of visual incongruence. We propose an explanation of these findings that draws on the cue integration theory of SoA. We also discuss the implications of these findings for HCI research and design

가상현실 내 정보 불일치를 활용한 인지기능 평가: 탐색적 고찰

학위논문(박사) -- 서울대학교대학원 : 인문대학 협동과정 인지과학전공, 2022.2. 이경민.본 박사논문의 목적은 가상현실 내에서 발생하는 정보불일치에 대해서 알아보고, 정보 불일치로 인한 인지적 반응을 인지기능 평가에 활용할 수 있는 방안을 고찰하고자 함이다. 가상현실 주방과제를 구현하여 과제 수행 중 나타나는 움직임과 인지작용의 특성을 알아보고자 하였다. 또한 VR에서 과제수행 시 나타나는 인지 부하의 요인을 탐색하였다. 특히, 감각운동 조절 측면에서 가상현실 내 발생하는 정보불일치로 인한 인지 과부하를 살펴보았다. 첫째, 가상현실과 실제환경에서 작동하는 인지과정이 어떻게 다른지 알아보기 위해 두 환경 간의 과제 수행 차이를 비교하였다. 젊은 성인 그룹에서는 어려운 주방과제 수행 시 가상현실과 실제환경 간의 수행시간에 유의한 차이가 있었지만 쉬운 주방 과제에서는 차이가 없었다. 반면 노인 집단에서는 과제의 난이도와 관계없이 두 환경 간의 수행 시간에 상당한 차이가 있었다. 노인의 경우 가상현실에서 감각운동 조절의 어려움을 보였다. 즉 노인의 경우 젊은 성인에 비해 가상현실 내에서의 감각운동 조절이 더 어렵기 때문에 이로 인한 인지적 부하가 과제 수행 자체의 인지적 부하에 가중되어 과제 난이도가 어려워지면 인지용량의 한계를 초과하게 된다. 둘째, 가상 주방과제 수행 시 인지기능이 저하됨에 따라 갑자기 휙 움직이는(jerky) 패턴을 보이는 것으로 나타났다. 이는 인지기능이 저하된 노인의 경우 환경에 대한 예측력이 저하되어 최소 저크운동 조절(minimal jerk movement control)에 어려움이 있음을 시사한다. 또한 인지기능이 높은 그룹보다 인지기능이 낮은 노인 그룹의 경우 과제가 완료될 때까지의 일련의 움직임 단계가 더 많았다. 인지기능이 저하됨에 따라 비효율적이고 분주한 움직임을 보인다고 할 수 있다. 또한 다중회귀분석 결과, 노인이 가상현실 주방과제를 효율적으로 수행함에 있어 연령 및 학력 보다는 인지기능이 가장 영향을 미치는 요인으로 나타났다. 즉 가상현실 기반 과제수행은 순수 인지기능만을 평가하는 새로운 대안으로 제시할 수 있다. 마지막으로 감각운동 피드백의 예측불가능성(unpredictability)이 가상현실에서 인지부하를 유발하는 방식을 알아보고자 하였다. 섭동의 예측 가능성에 따른 반응 시간과 이동 속도를 암묵적 5°와 명시적 15° 섭동 조건에서 각각 측정하였다. 그 결과 암묵적 운동 제어 시 섭동의 변화를 예측할 수 없을 때 움직임의 정확도를 높이기 위해 움직임이 느려지는 전략(accuracy and speed trade-off)을 사용하는 것으로 나타났다. 즉, 감각운동조절 과정 상에서 정보 불일치로 인한 예측 불가능성에 대해 우리의 뇌는 다른 인지전략을 취한다고 설명할 수 있다. 결론적으로 가상현실은 기술적 충실도(fidelity) 문제로 인해 감각 피드백이 예측 불가능하고 가변적이기 때문에 실제 환경보다 더 많은 인지 부하를 유발한다. 특히 가상현실에서의 감각운동 조절은 실제환경에서 인간의 운동 시스템이 적응된 방식과는 다르다고 볼 수 있다. 즉 가상현실 내에서는 감각운동 시스템이 예측할 수 없는 환경에 적응하기 위해 다른 인지 전략을 취하게 된다. 환경에 따른 효율적인 인지전략의 전환은 중앙 집행기능(central executive)과 관련 있으며, 이러한 특징을 활용한 가상현실기반 과제는 새로운 인지기능 평가의 대안으로 제시할 수 있다.The purpose of this dissertation was to investigate information mismatch in virtual reality (VR) and explore the possibility of using the cognitive reaction arising from information mismatch for cognitive evaluation. The virtual kitchen task was used to observe the subjects’ behaviors while performing the task, and to investigate the characteristics of movement and cognitive processes appearing during the performance of the virtual task. In addition, an attempt was made to explore the factors of cognitive overload in VR that determine the difference compared to a performance in the real environment. In particular, this study aimed to investigate how information mismatch occurring in VR causes cognitive overload in terms of sensorimotor control. First, it questioned how the cognitive process in VR differs from the real environment and also investigated the factors affecting the performance of tasks in VR. In the young adult group, while there was a significant difference between the execution time in VR and in the real environment in the difficult kitchen task, there was no such difference in the easy kitchen task. Meanwhile, among the elderly, there was a significant difference between the execution time in VR and in the real environment regardless of whether the task was difficult or easy. It was thought that cognitive load was caused due to difficulties in sensorimotor control in VR. It was found that the cognitive capacity is challenged when the task is difficult because the load of task performance itself and the load of sensorimotor control are doubling. Second, it was found that as the cognitive function decreased, an abrupt and jerky movement pattern was exhibited during the virtual kitchen task. The number of sequences in movement until the task was completed was also busier in the elderly group with lower cognitive function in contrast with those with higher cognitive function. In the case of the elderly with deteriorated cognitive function, it is suggested that there is difficulty in minimal jerk movement control because the predictive ability responding to environment is decreased. In addition, according to the results of multiple regression, cognitive function of the elderly is the most influential factor in performing VR tasks, other than age and educational background, which means that purely evaluating cognitive function may be suggested. Third, an attempt was made to verify how the unpredictability of sensorimotor feedback causes cognitive load in VR. The reaction time and speed of movement depending on the predictability of perturbation were measured in implicit 5 degrees and explicit 15 degrees perturbation. When the subject was unable to predict the variation of perturbation only in implicit motor control, reaching became slower and it took more time due to the accuracy and speed trade-off. In other words, unpredictability due to information mismatch leads to the use of different cognitive strategies in brain mechanisms. In conclusion, VR induces more cognitive load than the real environment because the sensory feedback is unpredictable and variable due to technical fidelity problems. The sensorimotor control in VR is challenged by the way the human motor system is adapted. Further, it was found that an unpredictable environment requires different cognitive strategies for the sensorimotor system to adapt to it. The manner in which effective cognitive strategies are taken represents an efficient central executive function. From this perspective, VR-based cognitive evaluation, using such attributes, is thought to be an alternative method for early screening of cognitive decline.Chapter 1. Introduction 7 1.1 Research motivation and introductory overview 7 1.2 Research goal and questions 7 1.2.1 Overall research goal 7 1.2.2 Research questions 8 1.2.3 Research contributions 8 1.3 Thesis structure 8 Chapter 2. Literature Review 10 2.1 Virtual Reality (VR) as ecological method for cognitive evaluation 10 2.2 Sub-types of VR based tasks according to target cognitive function 12 2.2.1. VR task for spatial navigation 13 2.2.2. VR task for memory 14 2.2.3. VR task for executive function 16 2.3 Factors affecting on VR performance 19 2.3.1. General 19 2.3.2. Age effects on VR performance 20 2.3.3. Cognitive challenges in VR 21 2.3.4. Feasibility of VR task for dementia 22 2.4 Cognitive load in VR 23 2.4.1. Immersive versus non-immersive VR 23 2.4.2. Sense of presence and situated cognition 26 2.4.3. Sensorimotor adaptation in VR 28 2.5 Sensorimotor control in VR 29 2.5.1 Predictive brain and internal model for motor control 29 2.5.2 Explicit and implicit process in motor control 31 2.5.3 Accuracy & speed tradeoff in cognitive control 31 2.6 Executive control for information mismatch in information processing 32 Chapter 3. Differences in Cognitive Load Between Real and VR Environment 34 3.1 Introduction 34 3.2 Method 37 3.3 Results 40 3.4 Discussion 45 Chapter 4. The Efficiency of Movement Trajectory and Sequence in VR According to Cognitive Function in the Elderly 50 4.1 Introduction 50 4.2 Method 52 4.3 Results 53 4.4 Discussion 56 Chapter 5. Factors that Affect the Performance of Immersive Virtual Kitchen Tasks in the Elderly 59 5.1 Introduction 59 5.2 Method 62 5.3 Results 64 5.4 Discussion 70 Chapter 6. Effect of Predictability of Sensorimotor Feedback on Cognitive Load in VR 74 6.1 Introduction 74 6.2 Method 77 6.3 Results 79 6.4 Discussion 84 Chapter 7. Conclusion 88 7.1 Summary of findings 88 7.2 Future direction of research 90 References 92박

Congruency of Information Rather Than Body Ownership Enhances Motor Performance in Highly Embodied Virtual Reality

In immersive virtual reality, the own body is often visually represented by an avatar. This may induce a feeling of body ownership over the virtual limbs. Importantly, body ownership and the motor system share neural correlates. Yet, evidence on the functionality of this neuroanatomical coupling is still inconclusive. Findings from previous studies may be confounded by the congruent vs. incongruent multisensory stimulation used to modulate body ownership. This study aimed to investigate the effect of body ownership and congruency of information on motor performance in immersive virtual reality. We aimed to modulate body ownership by providing congruent vs. incongruent visuo-tactile stimulation (i.e., participants felt a brush stroking their real fingers while seeing a virtual brush stroking the same vs. different virtual fingers). To control for congruency effects, unimodal stimulation conditions (i.e., only visual or tactile) with hypothesized low body ownership were included. Fifty healthy participants performed a decision-making (pressing a button as fast as possible) and a motor task (following a defined path). Body ownership was assessed subjectively with established questionnaires and objectively with galvanic skin response (GSR) when exposed to a virtual threat. Our results suggest that congruency of information may decrease reaction times and completion time of motor tasks in immersive virtual reality. Moreover, subjective body ownership is associated with faster reaction times, whereas its benefit on motor task performance needs further investigation. Therefore, it might be beneficial to provide congruent information in immersive virtual environments, especially during the training of motor tasks, e.g., in neurorehabilitation interventions

Recent developments in biofeedback for neuromotor rehabilitation

The original use of biofeedback to train single muscle activity in static positions or movement unrelated to function did not correlate well to motor function improvements in patients with central nervous system injuries. The concept of task-oriented repetitive training suggests that biofeedback therapy should be delivered during functionally related dynamic movement to optimize motor function improvement. Current, advanced technologies facilitate the design of novel biofeedback systems that possess diverse parameters, advanced cue display, and sophisticated control systems for use in task-oriented biofeedback. In light of these advancements, this article: (1) reviews early biofeedback studies and their conclusions; (2) presents recent developments in biofeedback technologies and their applications to task-oriented biofeedback interventions; and (3) discusses considerations regarding the therapeutic system design and the clinical application of task-oriented biofeedback therapy. This review should provide a framework to further broaden the application of task-oriented biofeedback therapy in neuromotor rehabilitation