Building trust in autonomous vehicles: Role of virtual reality driving simulators in HMI design

The investigation of factors contributing at making humans trust Autonomous Vehicles (AVs) will play a fundamental role in the adoption of such technology. The user's ability to form a mental model of the AV, which is crucial to establish trust, depends on effective user-vehicle communication; thus, the importance of Human-Machine Interaction (HMI) is poised to increase. In this work, we propose a methodology to validate the user experience in AVs based on continuous, objective information gathered from physiological signals, while the user is immersed in a Virtual Reality-based driving simulation. We applied this methodology to the design of a head-up display interface delivering visual cues about the vehicle' sensory and planning systems. Through this approach, we obtained qualitative and quantitative evidence that a complete picture of the vehicle's surrounding, despite the higher cognitive load, is conducive to a less stressful experience. Moreover, after having been exposed to a more informative interface, users involved in the study were also more willing to test a real AV. The proposed methodology could be extended by adjusting the simulation environment, the HMI and/or the vehicle's Artificial Intelligence modules to dig into other aspects of the user experience

Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using EEG

With the recent surge of affordable, high-performance virtual reality (VR) headsets, there is unlimited potential for applications ranging from education, to training, to entertainment, to fitness and beyond. As these interfaces continue to evolve, passive user-state monitoring can play a key role in expanding the immersive VR experience, and tracking activity for user well-being. By recording physiological signals such as the electroencephalogram (EEG) during use of a VR device, the user\u27s interactions in the virtual environment could be adapted in real-time based on the user\u27s cognitive state. Current VR headsets provide a logical, convenient, and unobtrusive framework for mounting EEG sensors. The present study evaluates the feasibility of passively monitoring cognitive workload via EEG while performing a classical n-back task in an interactive VR environment. Data were collected from 15 participants and the spatio-spectral EEG features were analyzed with respect to task performance. The results indicate that scalp measurements of electrical activity can effectively discriminate three workload levels, even after suppression of a co-varying high-frequency activity

Assessing Executive Function Impairments and Comorbidity between ADHD and Stuttering

Stuttering and ADHD are often considered ‘comorbid’ because different types of symptoms and processing issues occur in, for example, fluency, attention and working memory. This thesis addresses whether or not these shared factors signify fundamental similarities between stuttering and ADHD that distinguish them from typical controls. This is done in two main ways: First, a comparison is made of details of performance on attention capabilities using a range of behavioural and physiological measures in various test environments, including Web and VR approaches; Second, modelling analyses are conducted that compare networks representing participants’ performance across groups. Using the Load Theory of Attention methodology (Chapter 2), which addresses how to focus attention and ignore distractions up to a point where load exceeds perceptual capacity, it was observed that the performance of participants who stutter was significantly lower from the performance of controls in the auditory selective and divided attention tasks. The results showed that tasks in which attention demands enhanced were effective in detecting limitations in audio processing by PWS. Extending the task in the visual, audio and audio-visual domains in a virtual reality environment in people who stutter, (PWS) as well as people with ADHD (PWADHD) it was found that while audio was more affected in PWS, audio and audio-visual domains were affected in PWADHD. Lastly, Network Models (NMs) from the measures examined showed that comorbidity between PWS and PWADHD is limited. For better clinical assessments of attention, fluency and working memory problems, a Linear Mixed Model (LMM) was included in chapter 3 to understand if gender imbalance affected the results of PWS, PWADHD and controls in a selective attention task. LMM correctly determined that the gender imbalance did not affect the participants performance and PWS performed significantly worse from PWADHD showing that the groups were not comorbid and PWS is impaired in selective attention tasks. Further investigations were made in chapter 4 on PWS, PWADHD and controls in which data collected was extended to behavioural as well as physiological measures in a selective attention task implemented in a Virtual Reality (VR) environment. Although both PWS and PWADHD differed from controls with lower performance on the task, impulsive behaviours were only present in PWADHD (higher NOF) while inattentiveness was observed only in PWS (lower FD, higher theta activity). The architecture of NMs was different between PWS and PWADHD in task performance confirming again that comorbidity between groups is overstated while the frontal cortex is impaired in both groups as shown by NMs from EEG measures. While previous chapters showed that selective and divided attention tasks in Executive Function (EF) can correctly assess attention problems in PWS, chapters 5, 6 and 7 aimed at understanding which attention type in EF is impaired in PWADHD and can correctly assess attention problems in this group. An extensive investigation was made from 10 VR tasks that drew upon different attention types on behavioural measures and responses from questionnaires (chapter 5), eye measures (chapter 6) and brain activity (chapter 7). PWADHD were compared to controls on all the measures. NMs showed that sustained attention tasks in all domains and switched attention task only in the visual domain assessed ADHD traits in PWADHD on the measures examined. Furthermore, prefrontal cortex was impaired as shown from NMs in EEG measures. Finally, NMs were compared between controls, PWADHD and PWS in chapter 8 on cognitive factors including attention, fluency and working memory. NMs confirmed previous findings that the comorbidity of symptoms of both disorders is overstated. NM architecture between controls and PWADHD was similar, but both differed from PWS. Working memory was a strong factor that affected attention in all groups but the way it affected attention differed between PWS and PWADHD

Attention-Based Applications in Extended Reality to Support Autistic Users: A Systematic Review

With the rising prevalence of autism diagnoses, it is essential for research to understand how to leverage technology to support the diverse nature of autistic traits. While traditional interventions focused on technology for medical cure and rehabilitation, recent research aims to understand how technology can accommodate each unique situation in an efficient and engaging way. Extended reality (XR) technology has been shown to be effective in improving attention in autistic users given that it is more engaging and motivating than other traditional mediums. Here, we conducted a systematic review of 59 research articles that explored the role of attention in XR interventions for autistic users. We systematically analyzed demographics, study design and findings, including autism screening and attention measurement methods. Furthermore, given methodological inconsistencies in the literature, we systematically synthesize methods and protocols including screening tools, physiological and behavioral cues of autism and XR tasks. While there is substantial evidence for the effectiveness of using XR in attention-based interventions for autism to support autistic traits, we have identified three principal research gaps that provide promising research directions to examine how autistic populations interact with XR. First, our findings highlight the disproportionate geographic locations of autism studies and underrepresentation of autistic adults, evidence of gender disparity, and presence of individuals diagnosed with co-occurring conditions across studies. Second, many studies used an assortment of standardized and novel tasks and self-report assessments with limited tested reliability. Lastly, the research lacks evidence of performance maintenance and transferability.Comment: [Accepted version] K. Wang, S. J. Julier and Y. Cho, "Attention-Based Applications in Extended Reality to Support Autistic Users: A Systematic Review," in IEEE Access, vol. 10, pp. 15574-15593, 2022, doi: 10.1109/ACCESS.2022.314772

Evaluation of Detecting Cybersickness via VR HMD Positional Measurements Under Realistic Usage Conditions.

With the resurgence of virtual reality, head-mounted displays (VR HMD) technologies since 2015, VR technology is becoming ever more present in people's day-to-day lives. However, one significant barrier to this progress is a condition called cybersickness, a form of motion sickness induced by the usage of VR HMD’s. It is often debilitating to sufferers, resulting in symptoms anywhere from mild discomfort to full-on vomiting. Much research effort focuses on identifying the cause of and solution to this problem, with many studies reporting various factors that influence cybersickness, such as vection and field of view. However, there is often disagreement in these studies' results and comparing the results is often complicated as stimuli used for the experiments vary wildly. This study theorised that these results' mismatch might partially be down to the different mental loads of these tasks, which may influence cybersickness and stability-based measurement methods such as postural stability captured by the centre of pressure (COP) measurements. One recurring desire in these research projects is the idea of using the HMD device itself to capture the stability of the users head. However, measuring the heads position via the VR HMD is known to have inaccuracies meaning a perfect representation of the heads position cannot be measured. This research took the HTC Vive headset and used it to capture the head position of multiple subjects experiencing two different VR environments under differing levels of cognitive load. The design of these test environments reflected normal VR usage. This research found that the VR HMD measurements in this scenario may be a suitable proxy for recording instability. However, the underlying method was greatly influenced by other factors, with cognitive load (5.4% instability increase between the low and high load conditions) and test order (2.4% instability decrease between first run and second run conditions) having a more significant impact on the instability recorded than the onset of cybersickness (2% instability increase between sick and well participants). Also, separating participants suffering from cybersickness from unaffected participants was not possible based upon the recorded motion alone. Additionally, attempts to capture stability data during actual VR gameplay in specific areas of possible head stability provided mixed results and failed to identify participants exhibiting symptoms of cybersickness successfully. In conclusion, this study finds that while a proxy measurement for head stability is obtainable from an HTC Vive headset, the results recorded in no way indicate cybersickness onset. Additionally, the study proves cognitive load and test order significantly impact stability measurements recorded in this way. As such, this approach would need calibration on a case-by-case basis if used to detect cybersickness

An Analysis of Physiological and Psychological Responses in Virtual Reality and Flat Screen Gaming

Recent research has focused on the effectiveness of Virtual Reality (VR) in games as a more immersive method of interaction. However, there is a lack of robust analysis of the physiological effects between VR and flatscreen (FS) gaming. This paper introduces the first systematic comparison and analysis of emotional and physiological responses to commercially available games in VR and FS environments. To elicit these responses, we first selected four games through a pilot study of 6 participants to cover all four quadrants of the valence-arousal space. Using these games, we recorded the physiological activity, including Blood Volume Pulse and Electrodermal Activity, and self-reported emotions of 33 participants in a user study. Our data analysis revealed that VR gaming elicited more pronounced emotions, higher arousal, increased cognitive load and stress, and lower dominance than FS gaming. The Virtual Reality and Flat Screen (VRFS) dataset, containing over 15 hours of multimodal data comparing FS and VR gaming across different games, is also made publicly available for research purposes. Our analysis provides valuable insights for further investigations into the physiological and emotional effects of VR and FS gaming.Comment: This work has been submitted to the IEEE Transactions on Affective Computing for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessibl

The Effect of Augmented Reality Treatment on Learning, Cognitive Load, and Spatial Visualization Abilities

This study investigated the effects of Augmented Reality (AR) on learning, cognitive load and spatial abilities. More specifically, it measured learning gains, perceived cognitive load, and the role spatial abilities play with students engaged in an astronomy lesson about lunar phases. Research participants were 182 students from a public university in southeastern United States, and were recruited from psychology research pool. Participants were randomly assigned to two groups: (a) Augmented Reality and Text Astronomy Treatment (ARTAT); and (b) Images and Text Astronomy Treatment (ITAT). Upon entering the experimental classroom, participants were given (a) Paper Folding Test to measure their spatial abilities; (b) the Lunar Phases Concept Inventory (LPCI) pre-test; (c) lesson on Lunar Phases; (d) NASA-TLX to measure participants’ cognitive load; and (e) LPCI post-test. Statistical analysis found (a) no statistical difference for learning gains between the ARTAT and ITAT groups; (b) statistically significant difference for cognitive load; and (c) no significant difference for spatial abilities scores