Workload alerts - using physiological measures of mental workload to provide feedback during tasks

Feedback is valuable for allowing us to improve on tasks. While retrospective feedback can help us improve for next time, feedback “in action” can allow us to improve the outcome of on-going tasks. In this paper, we use data from functional Near InfraRed Spectroscopy to provide participants with feedback about their Mental Workload levels during high-workload tasks. We evaluate the impact of this feedback on task performance and perceived task performance, in comparison to industry standard mid-task self assessments, and explore participants’ perceptions of this feedback. In line with previous work, we confirm that deploying self-reporting methods affect both perceived and actual performance. Conversely, we conclude that our objective concurrent feedback correlated more closely with task demand, supported reflection in action, and did not negatively affect performance. Future work, however, should focus on the design of this feedback and the potential behaviour changes that will result

Tracking team mental workload by multimodal measurements in the operating room

Mental workload and its effects on surgical performance are underexplored topics, despite their importance for operating room (OR) efficiency and patient safety. We developed a multimodal platform that can simultaneously collect data from EEG, heart rate and breathing rate, tool handle pressure, and eye tracker from mobile subjects. We performed experiments using the Fundamentals of Laparoscopic Surgery model, with 22 subjects of varying skill levels ranging from nonsurgeon to expert. The results indicated significant modulations of the measurements depending on pupil size, heart rate variability, P300 response, tool pressure, task difficulty, time-on-task, and skill level. These provide evidence that physiology based metrics can be used in automated classification of fine gradations of skill, the assessment and certification of surgery trainees, developing real-time flags and warnings for the OR, and validating new OR technology

The psychology of driving automation: A discussion with Professor Don Norman

Introducing automation into automobiles had inevitable consequences for the driver and driving. Systems that automate longitudinal and lateral vehicle control may reduce the workload of the driver. This raises questions of what the driver is able to do with this 'spare' attentional capacity. Research in our laboratory suggests that there is unlikely to be any spare capacity because the attentional resources are not 'fixed'. Rather, the resources are inextricably linked to task demand. This paper presents some of the arguments for considering the psychological aspects of the driver when designing automation into automobiles. The arguments are presented in a conversation format, based on discussions with Professor Don Norman. Extracts from relevant papers to support the arguments are presented

Analysis and evaluation of the pilot attentional model

Pendant les opérations de vol, les pilotes sont exposés à une variété de conditions émotionnelles, mentales et physiques qui peuvent affecter leurs performances et leur attention. Par conséquent, il est crucial de surveiller leur charge de travail et leurs niveaux d'attention pour maintenir la sécurité et l'efficacité de l'aviation, notamment dans les situations d'urgence. La charge de travail fait référence aux exigences cognitives et physiques imposées aux pilotes lors d'un vol. Des niveaux élevés de charge de travail peuvent entraîner une fatigue mentale, une attention réduite et une surcharge cognitive, ce qui peut entraver leur capacité à effectuer leurs tâches de manière efficace et efficiente. L'attention est un processus cognitif complexe qui limite la capacité de se concentrer et de comprendre tout en même temps. Dans les tâches de traitement de l'information visuelle, la vision humaine est la principale source du mécanisme d'attention visuelle. Le mode de distribution de l'attention d'un pilote a un impact significatif sur la quantité d'informations qu'il acquiert, car la vision est le canal le plus critique pour l'acquisition d'informations. Une mauvaise allocation des ressources attentionnelles peut amener les pilotes à négliger ou à oublier des paramètres spécifiques, ce qui entraîne des risques graves pour la sécurité des aéronefs. Ainsi, cette étude vise à étudier les niveaux d'attention des pilotes lors d'une procédure de décollage simulée, en mettant l'accent particulièrement sur les périodes critiques telles que les pannes de moteur. Pour ce faire, l'étude examine s'il existe une corrélation entre la dilatation de la pupille, mesurée à l'aide de la technologie de suivi oculaire, et les niveaux d'engagement, mesurés à l'aide de l'EEG. Les résultats indiquent que les changements de taille de la pupille sont effectivement corrélés aux changements d'activité de l'EEG, suggérant que la dilatation de la pupille peut être utilisée comme un indicateur fiable de l'engagement et de l'attention. Sur la base de ces résultats, la dilatation de la pupille et l'EEG peuvent être utilisés en combinaison pour examiner de manière globale le comportement des pilotes, car les deux mesures sont des indicateurs valides de l'engagement et de la charge cognitive. De plus, l'utilisation de ces mesures peut aider à identifier les périodes critiques où les niveaux d'attention des pilotes nécessitent une surveillance étroite pour garantir la sécurité et l'efficacité de l'aviation. Cette étude met en évidence l'importance de surveiller la charge de travail et les niveaux d'attention des pilotes et recommande d'utiliser les mesures de dilatation de la pupille et d'EEG pour évaluer la charge cognitive et l'engagement d'un pilote pendant les opérations de vol, améliorant ainsi la sécurité et l'efficacité de l'aviation.During flight operations, pilots are exposed to a variety of emotional, mental, and physical conditions that can affect their performance and attention. Therefore, it is crucial to monitor their workload and attention levels to maintain aviation safety and efficiency, particularly in emergency situations. Workload refers to the cognitive and physical demands placed on pilots during a flight. High levels of workload can lead to mental fatigue, reduced attention, and cognitive overload, which can hinder their ability to perform their tasks effectively and efficiently. Attention is a complex cognitive process that limits the ability to focus and comprehend everything simultaneously. In visual information processing tasks, human vision is the primary source of the visual attention mechanism. A pilot's attention distribution mode significantly impacts the amount of information they acquire, as vision is the most critical channel for information acquisition. Improper allocation of attention resources can cause pilots to overlook or forget specific parameters, resulting in severe risks to aircraft safety. Thus, this study aims to investigate pilots' attention levels during a simulated takeoff procedure, with a specific focus on critical periods such as engine failures. To achieve this, the study examines whether there is a correlation between pupil dilation, measured using eye-tracking technology, and engagement levels, measured using EEG. The results indicate that changes in pupil size are indeed correlated with changes in EEG activity, suggesting that pupil dilation can be used as a reliable indicator of engagement and attention. Based on these findings, pupil dilation and EEG can be used in combination to comprehensively examine pilot behavior since both measures are valid indicators of engagement and cognitive workload. Furthermore, using these measures can help identify critical periods where pilots' attention levels require close monitoring to ensure aviation safety and efficiency. This study emphasizes the significance of monitoring pilots' workload and attention levels and recommends using pupil dilation and EEG measures to assess a pilot's cognitive workload and engagement during flight operations, ultimately enhancing aviation safety and efficiency

Selecting Metrics to Evaluate Human Supervisory Control Applications

The goal of this research is to develop a methodology to select supervisory control metrics. This methodology is based on cost-benefit analyses and generic metric classes. In the context of this research, a metric class is defined as the set of metrics that quantify a certain aspect or component of a system. Generic metric classes are developed because metrics are mission-specific, but metric classes are generalizable across different missions. Cost-benefit analyses are utilized because each metric set has advantages, limitations, and costs, thus the added value of different sets for a given context can be calculated to select the set that maximizes value and minimizes costs. This report summarizes the findings of the first part of this research effort that has focused on developing a supervisory control metric taxonomy that defines generic metric classes and categorizes existing metrics. Future research will focus on applying cost benefit analysis methodologies to metric selection. Five main metric classes have been identified that apply to supervisory control teams composed of humans and autonomous platforms: mission effectiveness, autonomous platform behavior efficiency, human behavior efficiency, human behavior precursors, and collaborative metrics. Mission effectiveness measures how well the mission goals are achieved. Autonomous platform and human behavior efficiency measure the actions and decisions made by the humans and the automation that compose the team. Human behavior precursors measure human initial state, including certain attitudes and cognitive constructs that can be the cause of and drive a given behavior. Collaborative metrics address three different aspects of collaboration: collaboration between the human and the autonomous platform he is controlling, collaboration among humans that compose the team, and autonomous collaboration among platforms. These five metric classes have been populated with metrics and measuring techniques from the existing literature. Which specific metrics should be used to evaluate a system will depend on many factors, but as a rule-of-thumb, we propose that at a minimum, one metric from each class should be used to provide a multi-dimensional assessment of the human-automation team. To determine what the impact on our research has been by not following such a principled approach, we evaluated recent large-scale supervisory control experiments conducted in the MIT Humans and Automation Laboratory. The results show that prior to adapting this metric classification approach, we were fairly consistent in measuring mission effectiveness and human behavior through such metrics as reaction times and decision accuracies. However, despite our supervisory control focus, we were remiss in gathering attention allocation metrics and collaboration metrics, and we often gathered too many correlated metrics that were redundant and wasteful. This meta-analysis of our experimental shortcomings reflect those in the general research population in that we tended to gravitate to popular metrics that are relatively easy to gather, without a clear understanding of exactly what aspect of the systems we were measuring and how the various metrics informed an overall research question

Measuring mental workload in IIR user studies with fNIRS

Gathering neuro-physiological data during user studies, and analysing the continuous data they produce, typically involves making a tradeoff between detail and practical utility. is paper describes our long-term work-in-progress towards developing study protocols for using functional Near-InfraRed Spectroscopy (fNIRS) with the aim of finding the ideal balance in this tradeoff. Our results show that fNIRS can be easily used in normal IIR user study conditions, is tolerant of minor movement artefacts (including speaking), and can still determine mental workload differences between different user interfaces designed for the same task

Comparing Types Of Adaptive Automation Within A Multi-tasking Environment

Throughout the many years of research examining the various effects of automation on operator performance, stress, workload, etc., the focus has traditionally been on the level of automation, and the invocation methods used to alter it. The goal of the current study is to instead examine the utilization of various types of automation with the goal of better meeting the operator’s cognitive needs, thus improving their performance, workload, and stress. The task, control of a simulated unmanned robotic system, is designed to specifically stress the operator’s visual perception capabilities to a greater degree. Two types of automation are implemented to support the operator’s performance of the task: an auditory beep aid intended to support visual perception resources, and a driving aid automating control of the vehicle’s navigation, offloading physical action execution resources. Therefore, a comparison can be made between types of automation intended to specifically support the mental dimension that is under the greatest demand (the auditory beep) against those that do not (the driving automation). An additional evaluation is made to determine the benefit of adaptively adjusting the level of each type of automation based on the current level of task demand, as well as the influence of individual differences in personality. Results indicate that the use of the auditory beep aid does improve performance, but also increases Temporal Demand and Effort. Use of driving automation appears to disengage the operator from the task, eliciting a vigilance response. Adaptively altering the level of automation to meet task demands has a mixed effect on performance and workload (reducing both) when the auditory beep automation is used. However, adaptive driving automation is clearly detrimental, iv causing an increase in workload while decreasing performance. Higher levels of Neuroticism are related to poorer threat detection performance, but personality differences show no indication of moderating the effects of either of the experimental manipulations. The results of this study show that the type of automation implemented within an environment has a considerable impact on the operator, in terms of performance as well as cognitive/emotional stat

Aerospace Medicine and Biology: A continuing bibliography, supplement 191

A bibliographical list of 182 reports, articles, and other documents introduced into the NASA scientific and technical information system in February 1979 is presented

How stress affects functional near-infrared spectroscopy (fNIRS) measurements of mental workload

Recent work has demonstrated that functional Near-Infrared Spectroscopy has the potential to measure changes in Mental Workload with increasing ecological validity. It is not clear, however, whether these measurements are affected by anxiety and stress of the workload, where our informal observations see some participants enjoying the workload and succeeding in tasks, while others worry and struggle with the tasks. This research evaluated the effects of stress on fNIRS measurements and performance, using the Montreal Imaging Stress Task to manipulate the experience of stress. While our results largely support this hypothesis, our conclusions were undermined by data from the Rest condition, which indicated that Mental Workload and Stress were often higher than during tasks. We hypothesize that participants were experiencing anxiety in anticipation of subsequent stress tasks. We discuss this hypothesis and present a revised study designed to better control for this result

Assessing the Impact of Haptic Peripheral Displays for UAV Operators

Objectives: A pilot study was conducted to investigate the effectiveness of continuous haptic peripheral displays in supporting multiple UAV supervisory control. Background: Previous research shows that continuous auditory peripheral displays can enhance operator performance in monitoring events that are continuous in nature, such as monitoring how well UAVs stay on their pre-planned courses. This research also shows that auditory alerts can be masked by other auditory information. Command and control operations are generally performed in noisy environments with multiple auditory alerts presented to the operators. In order to avoid this masking problem, another potentially useful sensory channel for providing redundant information to UAV operators is the haptic channel. Method: A pilot experiment was conducted with 13 participants, using a simulated multiple UAV supervisory control task. All participants completed two haptic feedback conditions (continuous and threshold), where they received alerts based on UAV course deviations and late arrivals to targets. Results: Threshold haptic feedback was found to be more effective for late target arrivals, whereas continuous haptic feedback resulted in faster reactions to course deviations. Conclusions: Continuous haptic feedback appears to be more appropriate for monitoring events that are continuous in nature (i.e., how well a UAV keeps its course). In contrast, threshold haptic feedback appears to better support response to discrete events (i.e., late target arrivals). Future research: Because this is a pilot study, more research is needed to validate these preliminary findings. A direct comparison between auditory and haptic feedback is also needed to provide better insights into the potential benefits of multi-modal peripheral displays in command and control of multiple UAVs.Prepared for Charles River Analytics, Inc