Neuronal correlates of continuous manual tracking under varying visual movement feedback in a virtual reality environment

To accurately guide one's actions online, the brain predicts sensory action feedback ahead of time based on internal models, which can be updated by sensory prediction errors. The underlying operations can be experimentally investigated in sensorimotor adaptation tasks, in which moving under perturbed sensory action feedback requires internal model updates. Here we altered healthy participants’ visual hand movement feedback in a virtual reality setup, while assessing brain activity with functional magnetic resonance imaging (fMRI). Participants tracked a continually moving virtual target object with a photorealistic, three-dimensional (3D) virtual hand controlled online via a data glove. During the continuous tracking task, the virtual hand's movements (i.e., visual movement feedback) were repeatedly periodically delayed, which participants had to compensate for to maintain accurate tracking. This realistic task design allowed us to simultaneously investigate processes likely operating at several levels of the brain's motor control hierarchy. FMRI revealed that the length of visual feedback delay was parametrically reflected by activity in the inferior parietal cortex and posterior temporal cortex. Unpredicted changes in visuomotor mapping (at transitions from synchronous to delayed visual feedback periods or vice versa) activated biological motion-sensitive regions in the lateral occipitotemporal cortex (LOTC). Activity in the posterior parietal cortex (PPC), focused on the contralateral anterior intraparietal sulcus (aIPS), correlated with tracking error, whereby this correlation was stronger in participants with higher tracking performance. Our results are in line with recent proposals of a wide- spread cortical motor control hierarchy, where temporoparietal regions seem to evaluate visuomotor congruence and thus possibly ground a self-attribution of movements, the LOTC likely processes early visual prediction errors, and the aIPS computes action goal errors and possibly corresponding motor corrections

The Principles of Art Therapy in Virtual Reality

In recent years, the field of virtual reality (VR) has shown tremendous advancements and is utilized in entertainment, scientific research, social networks, artistic creation, as well as numerous approaches to employ VR for psychotherapy. While the use of VR in psychotherapy has been widely discussed, little attention has been given to the potential of this new medium for art therapy. Artistic expression in VR is a novel medium which offers unique possibilities, extending beyond classical expressive art mediums. Creation in VR includes options such as three-dimensional painting, an immersive creative experience, dynamic scaling, and embodied expression. In this perspective paper, we present the potentials and challenges of VR for art therapy and outline basic principles for its implementation. We focus on the novel qualities offered by this creative medium (the virtual environment, virtual materials, and unreal characteristics) and on the core aspects of VR (such as presence, immersivity, point of view, and perspective) for the practice of art therapy

Active inference under visuo-proprioceptive conflict: Simulation and empirical results

It has been suggested that the brain controls hand movements via internal models that rely on visual and proprioceptive cues about the state of the hand. In active inference formulations of such models, the relative influence of each modality on action and perception is determined by how precise (reliable) it is expected to be. The 'top-down' affordance of expected precision to a particular sensory modality is associated with attention. Here, we asked whether increasing attention to (i.e., the precision of) vision or proprioception would enhance performance in a hand-target phase matching task, in which visual and proprioceptive cues about hand posture were incongruent. We show that in a simple simulated agent-based on predictive coding formulations of active inference-increasing the expected precision of vision or proprioception improved task performance (target matching with the seen or felt hand, respectively) under visuo-proprioceptive conflict. Moreover, we show that this formulation captured the behaviour and self-reported attentional allocation of human participants performing the same task in a virtual reality environment. Together, our results show that selective attention can balance the impact of (conflicting) visual and proprioceptive cues on action-rendering attention a key mechanism for a flexible body representation for action

가상현실 내 정보 불일치를 활용한 인지기능 평가: 탐색적 고찰

학위논문(박사) -- 서울대학교대학원 : 인문대학 협동과정 인지과학전공, 2022.2. 이경민.본 박사논문의 목적은 가상현실 내에서 발생하는 정보불일치에 대해서 알아보고, 정보 불일치로 인한 인지적 반응을 인지기능 평가에 활용할 수 있는 방안을 고찰하고자 함이다. 가상현실 주방과제를 구현하여 과제 수행 중 나타나는 움직임과 인지작용의 특성을 알아보고자 하였다. 또한 VR에서 과제수행 시 나타나는 인지 부하의 요인을 탐색하였다. 특히, 감각운동 조절 측면에서 가상현실 내 발생하는 정보불일치로 인한 인지 과부하를 살펴보았다. 첫째, 가상현실과 실제환경에서 작동하는 인지과정이 어떻게 다른지 알아보기 위해 두 환경 간의 과제 수행 차이를 비교하였다. 젊은 성인 그룹에서는 어려운 주방과제 수행 시 가상현실과 실제환경 간의 수행시간에 유의한 차이가 있었지만 쉬운 주방 과제에서는 차이가 없었다. 반면 노인 집단에서는 과제의 난이도와 관계없이 두 환경 간의 수행 시간에 상당한 차이가 있었다. 노인의 경우 가상현실에서 감각운동 조절의 어려움을 보였다. 즉 노인의 경우 젊은 성인에 비해 가상현실 내에서의 감각운동 조절이 더 어렵기 때문에 이로 인한 인지적 부하가 과제 수행 자체의 인지적 부하에 가중되어 과제 난이도가 어려워지면 인지용량의 한계를 초과하게 된다. 둘째, 가상 주방과제 수행 시 인지기능이 저하됨에 따라 갑자기 휙 움직이는(jerky) 패턴을 보이는 것으로 나타났다. 이는 인지기능이 저하된 노인의 경우 환경에 대한 예측력이 저하되어 최소 저크운동 조절(minimal jerk movement control)에 어려움이 있음을 시사한다. 또한 인지기능이 높은 그룹보다 인지기능이 낮은 노인 그룹의 경우 과제가 완료될 때까지의 일련의 움직임 단계가 더 많았다. 인지기능이 저하됨에 따라 비효율적이고 분주한 움직임을 보인다고 할 수 있다. 또한 다중회귀분석 결과, 노인이 가상현실 주방과제를 효율적으로 수행함에 있어 연령 및 학력 보다는 인지기능이 가장 영향을 미치는 요인으로 나타났다. 즉 가상현실 기반 과제수행은 순수 인지기능만을 평가하는 새로운 대안으로 제시할 수 있다. 마지막으로 감각운동 피드백의 예측불가능성(unpredictability)이 가상현실에서 인지부하를 유발하는 방식을 알아보고자 하였다. 섭동의 예측 가능성에 따른 반응 시간과 이동 속도를 암묵적 5°와 명시적 15° 섭동 조건에서 각각 측정하였다. 그 결과 암묵적 운동 제어 시 섭동의 변화를 예측할 수 없을 때 움직임의 정확도를 높이기 위해 움직임이 느려지는 전략(accuracy and speed trade-off)을 사용하는 것으로 나타났다. 즉, 감각운동조절 과정 상에서 정보 불일치로 인한 예측 불가능성에 대해 우리의 뇌는 다른 인지전략을 취한다고 설명할 수 있다. 결론적으로 가상현실은 기술적 충실도(fidelity) 문제로 인해 감각 피드백이 예측 불가능하고 가변적이기 때문에 실제 환경보다 더 많은 인지 부하를 유발한다. 특히 가상현실에서의 감각운동 조절은 실제환경에서 인간의 운동 시스템이 적응된 방식과는 다르다고 볼 수 있다. 즉 가상현실 내에서는 감각운동 시스템이 예측할 수 없는 환경에 적응하기 위해 다른 인지 전략을 취하게 된다. 환경에 따른 효율적인 인지전략의 전환은 중앙 집행기능(central executive)과 관련 있으며, 이러한 특징을 활용한 가상현실기반 과제는 새로운 인지기능 평가의 대안으로 제시할 수 있다.The purpose of this dissertation was to investigate information mismatch in virtual reality (VR) and explore the possibility of using the cognitive reaction arising from information mismatch for cognitive evaluation. The virtual kitchen task was used to observe the subjects’ behaviors while performing the task, and to investigate the characteristics of movement and cognitive processes appearing during the performance of the virtual task. In addition, an attempt was made to explore the factors of cognitive overload in VR that determine the difference compared to a performance in the real environment. In particular, this study aimed to investigate how information mismatch occurring in VR causes cognitive overload in terms of sensorimotor control. First, it questioned how the cognitive process in VR differs from the real environment and also investigated the factors affecting the performance of tasks in VR. In the young adult group, while there was a significant difference between the execution time in VR and in the real environment in the difficult kitchen task, there was no such difference in the easy kitchen task. Meanwhile, among the elderly, there was a significant difference between the execution time in VR and in the real environment regardless of whether the task was difficult or easy. It was thought that cognitive load was caused due to difficulties in sensorimotor control in VR. It was found that the cognitive capacity is challenged when the task is difficult because the load of task performance itself and the load of sensorimotor control are doubling. Second, it was found that as the cognitive function decreased, an abrupt and jerky movement pattern was exhibited during the virtual kitchen task. The number of sequences in movement until the task was completed was also busier in the elderly group with lower cognitive function in contrast with those with higher cognitive function. In the case of the elderly with deteriorated cognitive function, it is suggested that there is difficulty in minimal jerk movement control because the predictive ability responding to environment is decreased. In addition, according to the results of multiple regression, cognitive function of the elderly is the most influential factor in performing VR tasks, other than age and educational background, which means that purely evaluating cognitive function may be suggested. Third, an attempt was made to verify how the unpredictability of sensorimotor feedback causes cognitive load in VR. The reaction time and speed of movement depending on the predictability of perturbation were measured in implicit 5 degrees and explicit 15 degrees perturbation. When the subject was unable to predict the variation of perturbation only in implicit motor control, reaching became slower and it took more time due to the accuracy and speed trade-off. In other words, unpredictability due to information mismatch leads to the use of different cognitive strategies in brain mechanisms. In conclusion, VR induces more cognitive load than the real environment because the sensory feedback is unpredictable and variable due to technical fidelity problems. The sensorimotor control in VR is challenged by the way the human motor system is adapted. Further, it was found that an unpredictable environment requires different cognitive strategies for the sensorimotor system to adapt to it. The manner in which effective cognitive strategies are taken represents an efficient central executive function. From this perspective, VR-based cognitive evaluation, using such attributes, is thought to be an alternative method for early screening of cognitive decline.Chapter 1. Introduction 7 1.1 Research motivation and introductory overview 7 1.2 Research goal and questions 7 1.2.1 Overall research goal 7 1.2.2 Research questions 8 1.2.3 Research contributions 8 1.3 Thesis structure 8 Chapter 2. Literature Review 10 2.1 Virtual Reality (VR) as ecological method for cognitive evaluation 10 2.2 Sub-types of VR based tasks according to target cognitive function 12 2.2.1. VR task for spatial navigation 13 2.2.2. VR task for memory 14 2.2.3. VR task for executive function 16 2.3 Factors affecting on VR performance 19 2.3.1. General 19 2.3.2. Age effects on VR performance 20 2.3.3. Cognitive challenges in VR 21 2.3.4. Feasibility of VR task for dementia 22 2.4 Cognitive load in VR 23 2.4.1. Immersive versus non-immersive VR 23 2.4.2. Sense of presence and situated cognition 26 2.4.3. Sensorimotor adaptation in VR 28 2.5 Sensorimotor control in VR 29 2.5.1 Predictive brain and internal model for motor control 29 2.5.2 Explicit and implicit process in motor control 31 2.5.3 Accuracy & speed tradeoff in cognitive control 31 2.6 Executive control for information mismatch in information processing 32 Chapter 3. Differences in Cognitive Load Between Real and VR Environment 34 3.1 Introduction 34 3.2 Method 37 3.3 Results 40 3.4 Discussion 45 Chapter 4. The Efficiency of Movement Trajectory and Sequence in VR According to Cognitive Function in the Elderly 50 4.1 Introduction 50 4.2 Method 52 4.3 Results 53 4.4 Discussion 56 Chapter 5. Factors that Affect the Performance of Immersive Virtual Kitchen Tasks in the Elderly 59 5.1 Introduction 59 5.2 Method 62 5.3 Results 64 5.4 Discussion 70 Chapter 6. Effect of Predictability of Sensorimotor Feedback on Cognitive Load in VR 74 6.1 Introduction 74 6.2 Method 77 6.3 Results 79 6.4 Discussion 84 Chapter 7. Conclusion 88 7.1 Summary of findings 88 7.2 Future direction of research 90 References 92박

The computational neurology of movement under active inference

We propose a computational neurology of movement based on the convergence of theoretical neurobiology and clinical neurology. A significant development in the former is the idea that we can frame brain function as a process of (active) inference, in which the nervous system makes predictions about its sensory data. These predictions depend upon an implicit predictive (generative) model used by the brain. This means neural dynamics can be framed as generating actions to ensure sensations are consistent with these predictions-and adjusting predictions when they are not. We illustrate the significance of this formulation for clinical neurology through simulating a clinical examination of the motor system; i.e. an upper limb coordination task. Specifically, we show how tendon reflexes emerge naturally under the right kind of generative model. Through simulated perturbations, pertaining to prior probabilities of this model's variables, we illustrate the emergence of hyperreflexia and pendular reflexes, reminiscent of neurological lesions in the corticospinal tract and cerebellum. We then turn to the computational lesions causing hypokinesia and deficits of coordination. This in silico lesion-deficit analysis provides an opportunity to revisit classic neurological dichotomies (e.g. pyramidal versus extrapyramidal systems) from the perspective of modern approaches to theoretical neurobiology-and our understanding of the neurocomputational architecture of movement control based on first principles

Neural correlates of performance monitoring during discrete and continuous tasks

Monitoring our actions is a key function of the brain for adaptive and successful behavior. Actions can be discrete such as when pressing a button, or continuous, such as driving a car. Moreover, we evaluate our actions as correct or erroneous (performance monitoring) and this appraisal of performance comes with various levels of confidence (metacognition). However, studies of performance monitoring have focused on discrete actions and are mostly agnostic to metacognitive judgments. The objective of this thesis was to extend the study of performance monitoring to more ecological conditions, in which monitoring occurs during continuous motor tasks under various degrees of error and confidence level. We first investigated the role of actions in performance monitoring together with metacognitive judgments, using simultaneous EEG and fMRI recordings. To dissociate the role of motor actions, we designed an experimental paradigm in which subjects had to rate their confidence level about an action that they had either performed themselves (a button press) based on a decision or passively observed (a virtual hand displayed). We found correlates of confidence in both condition, in the EEG and in the supplementary motor area (SMA). Furthermore, we found that subject showed better metacognitive performances when they were the agents of the action. This difference was further emphasized for subjects that showed higher activations of a network previously linked to motor inhibition and comprising the pre-SMA and inferior frontal gyrus (IFG). Our results imply that the SMA plays a primary role in the monitoring of performance, irrespectively of a commitment to a decision and the resulting action. Our findings also suggest that the additional neural processes leading to decisions and actions can inform the metacognitive judgments. In the following chapters, we ask whether electrophysiological correlates of performance monitoring can be found in less experimentally constrained paradigms for which motor output continuous unfolds and visual feedback is not segregated into discrete events. By decomposing the unfolding hand kinematics during a visuo-motor tracking task into periodic acceleration pulses âhenceforth referred to as sub-movements, we found three electrophysiological markers that could possibly be linked to performance monitoring. Firstly, we found an ERP in the SMA, time-locked to sub-movements which encoded the deviation of the hand, 110 ms before. Secondly, we found high-gamma activity in the ACC and SMA of epileptic patients, that was phase-locked to sub-movements. Thirdly, we found a transient modulation of mu oscillations over the ipsilateral sensorimotor cortices that depended on sub-movement amplitude. Altogether, these results provide a strong contribution in the understanding of the neurophysiological processes underlying performance monitoring. Our work proposes a methodological framework to study electrophysiological correlates of performance monitoring in less controlled paradigms during which continuous visual feedback has to be constantly integrated into motor corrections. In the conclusion chapter, we propose a way of extending current models of performance monitoring and decision making to explain the findings of this thesis by considering continuous motor tasks as a succession of decision making processes under time pressure and uncertainty

Investigating metabolic, vascular and structural neuroplasticity in healthy and diseased brain using advanced neuroimaging techniques

The brain’s lifelong capacity for reorganization is termed ‘plasticity’. It relies on molecular signalling translated into long lasting modifications. MRI has been widely used to assess neuroplasticity in vivo, showing brain’s ability to undergo functional and structural reorganization. However, there is a lack of understanding of the physiological events supporting neuroplasticity and advanced MRI techniques could help in the investigation of the biological meaning of these events and their alterations during neuroinflammation. This thesis has two main aims. Neuroscientifically, it aims to better understand mechanisms supporting neuroplasticity in the healthy and diseased brain. Methodologically, it aims to explore new MRI approaches to the study of neuroplasticity. The early experiments investigate the mechanisms underlying long-term neuroplasticity in MS. The studies then aim to elucidate the changes in brain energetics underlying adaptation in healthy and MS brain using calibrated fMRI. I explored new approaches to analyse the relative oxygen consumption during task adaptation in the same population. A new task to study short-term neuroplasticity was validated and used to demonstrate changes in resting blood flow after task execution. The same task was used to investigate the relationship between GM myelination and functional activity during task execution. Overall, we show the feasibility of using quantitative methods to study neuroplasticity, encouraging their application to improve biological interpretation in imaging studies. Our results highlight the importance of studying the brain as a network and the advantages of integrating different MRI modalities. We also show that our methods are applicable to MS populations, despite the observed metabolic impairment with neuroinflammation. Our methods may, in future, contribute to the study of disease progression and to the development of targeted interventions to limit the damage of inflammation