Detecting System Errors in Virtual Reality Using EEG Through Error-Related Potentials

When persons interact with the environment and experience or wit-ness an error (e.g. an unexpected event), a specific brain pattern,known as error-related potential (ErrP) can be observed in the elec-troencephalographic signals (EEG). Virtual Reality (VR) technologyenables users to interact with computer-generated simulated envi-ronments and to provide multi-modal sensory feedback. Using VRsystems can, however, be error-prone. In this paper, we investigatethe presence of ErrPs when Virtual Reality users face 3 types ofvisualization errors: (Te) tracking errors when manipulating virtualobjects, (Fe) feedback errors, and (Be) background anomalies. Weconducted an experiment in which 15 participants were exposed tothe 3 types of errors while performing a center-out pick and placetask in virtual reality. The results showed that tracking errors gener-ate error-related potentials, the other types of errors did not generatesuch discernible patterns. In addition, we show that it is possible todetect the ErrPs generated by tracking losses in single trial, with anaccuracy of 85%. This constitutes a first step towards the automaticdetection of error-related potentials in VR applications, paving theway to the design of adaptive and self-corrective VR/AR applicationsby exploiting information directly from the user’s brain

Electrocortical underpinnings of error monitoring in health and pathology

It becomes clear from the literature described above (Chapter 1), that the error monitoring mechanisms play a fundamental role in signalling the need for cognitive control. Many studies already provided a consistent evidence on the existence of peculiar ways in which the brain signals this need through electrophysiological changes. However, the following set of empirical studies aims to gain further insight into these complex processes by measuring brain activity changes in situations that alter the way one experience errors. The second Chapter (Chapter 2) consists of a brief commentary that was made in response to an article on the brain activity to action errors. In this commentary we propose new possibilities to explore our topic of interest, by taking advantage of EEG and modern virtual reality facilities. The thesis includes three EEG-VR studies: one on the error-mechanism in healthy participants (Chapter 3) and two studies on error monitoring system in pathological populations (Chapter 4, 5), as main parts of the core of the thesis. As a collateral project, in the Appendix, there is an EEG study on action observation in elite players (Chapter 7). In the first study (Chapter 3), we investigated a very simple but fundamental question. As we saw in the introduction, error-related signatures are evoked when an error occurs. But it is not clear how much of this is due to the occurrence of a violation of the intended goal or simply to the observation of a rare – thus less predictable – event. To this aim, we used a paradigm developed in the former years in our laboratory (Pavone et al., 2016; Spinelli et al., 2017), characterized by a setup in immersive Virtual Reality (VR) and simultaneous EEG recording. Building on the previous findings, we designed an EEG-VR study in which we manipulated the probability of observing errors in actions. In another study (Chapter 4) we investigated how erroneous actions are experienced by people with brain damage and diagnosis of Apraxia. Apraxic patients are people with hemispheric lesions and defective awareness on a variety of aspects that cover perceptuo-motor, cognitive or emotional domains. This study was developed after the results obtained by Canzano and colleagues (2014) in a behavioral study in which apraxic patients were asked to imitate the actions executed by the experimenter and judge their correctness; results revealed that bucco-facial apraxic patients manifest a specific deficit in detecting their own gestural errors when they are explicitly asked to judge them. With the present study we wanted to investigate apraxic brain’ response to action errors, while they embody an avatar in first person perspective (EEG-VR setup). The third study (Chapter 5) investigates the integrity of the error-monitoring system in Parkinson’s Disease and the impact of the dopaminergic treatment in the brain response to errors. To this aim we used the proposed VR action-observation paradigm, in which Parkinson patients observed successful and unsuccessful reach-to-grasp actions in first person perspective while EEG activity was recorded; the same patients were tested while being under dopaminergic treatment and during a dopaminergic withdrawal state. In another chapter we provide a critical overview of the findings of this work (General Discussion, Chapter 6). In the last chapter, the Appendix (Chapter 7), there is a collateral project of another research line of the Laboratory, in which I have being involved. In this study we are investigating the cortical underpinning of elite players during observation of goal-directed actions, in their domain of expertise. We recorded the EEG activity of elite wheelchair basketball players while observing free-throws performed by paraplegic athletes. We expected their brain correlates to be different from novice players and to be able to easily discriminate whether a basketball shot would be successful or unsuccessful (project still ongoing)

Midfrontal theta transcranial alternating current stimulation modulates behavioural adjustment after error execution

Cognitive control during conflict monitoring, error processing, and post-error adjustment appear to be associated with the occurrence of midfrontal theta (MFϴ). While this association is supported by correlational EEG studies, much less is known about the possible causal link between MFϴ and error and conflict processing. In the present study, we aimed to explore the role of band-specific effects in modulating the error system during a conflict resolution. In turn, we delivered transcranial alternating current stimulation (tACS) at different frequency bands (delta δ, theta θ, alpha α, beta β, gamma γ) and sham stimulation over the medial frontal cortex (MFC) in 36 healthy participants performing a modified version of the Flanker task. Task performance and reports about the sensations (e.g. visual flickering, cutaneous burning) induced by the different frequency bands, were also recorded. We found that online θ-tACS increased the response speed to congruent stimuli after error execution with respect to sham stimulation. Importantly, the accuracy following the errors did not decrease because of speed-accuracy trade off. Moreover, tACS evoked visual and somatosensory sensations were significantly stronger at α-tACS and β-tACS compared to other frequencies. Our findings suggest that theta activity plays a causative role in modulating behavioural adjustments during perceptual choices in a stimulus-response conflict task. © 2018 Federation of European Neuroscience Societies and John Wiley & Sons Lt

Performance monitoring during action observation and auditory lexical decisions

How does the brain monitor performances? Does expertise modulate this process? How does an observer’s error related activity differ from a performers own error related activity? How does ambiguity change the markers of error monitoring? In this thesis, I present two EEG studies and a commentary that sought to answer these questions. Both empirical studies concern performance monitoring in two different contexts and from two different personal perspectives, i.e. investigating the effects of expertise on electroencephalographic (EEG) neuromarkers of performance monitoring and in terms of monitoring own and others’ errors during actions and language processing. My first study focused on characterizing the electrophysiological responses in experts and control individuals while they are observing domain-specific actions in wheelchair basketball with correct and wrong outcomes (Chapter II). The aim of the commentary in the following chapter was to highlight the role of Virtual Reality approaches to error prediction during one’s own actions (Chapter III). The fourth chapter hypothesised that the error monitoring markers are present during both one’s own performance errors in a lexical decision task, and the observation of others’ performance errors (Chapter IV), however, the results suggested a further modulation of uncertainty created by our task design. The final chapter presents a general discussion that provides an overview of the results of my PhD work (Chapter V). The present chapter consists of a literature review in the leading frameworks of performance monitoring, action observation, visuo-motor expertise and language processing

The Impact of Hand Movement Velocity on Cognitive Conflict Processing in a 3D Object Selection Task in Virtual Reality.

Detecting and correcting incorrect body movements is an essential part of everyday interaction with one's environment. The human brain provides a monitoring system that constantly controls and adjusts our actions according to our surroundings. However, when our brain's predictions about a planned action do not match the sensory inputs resulting from that action, cognitive conflict occurs. Much is known about cognitive conflict in 1D/2D environments; however, less is known about the role of movement characteristics associated with cognitive conflict in 3D environment. Hence, we devised an object selection task in a virtual reality (VR) environment to test how the velocity of hand movements impacts human brain responses. From a series of analyses of EEG recordings synchronized with motion capture, we found that the velocity of the participants' hand movements modulated the brain's response to proprioceptive feedback during the task and induced a prediction error negativity (PEN). Additionally, the PEN originates in the anterior cingulate cortex and is itself modulated by the ballistic phase of the hand's movement. These findings suggest that velocity is an essential component of integrating hand movements with visual and proprioceptive information during interactions with real and virtual objects