Analysis of Software Design Patterns in Human Cognitive Performance Experiments

As Air Force operations continue to move toward the use of more autonomous systems and more human-machine teaming in general, there is a corresponding need to swiftly evaluate systems with these capabilities. We support this development through software design improvements of the execution of human cognitive performance experiments. This thesis sought to answer the following two research questions addressing the core functionality that these experiments rely on for execution and analysis: 1) What data infrastructure software requirements are necessary to execute the experimental design of human cognitive performance experiments? 2) How effectively does a central data mediator design pattern meet the time-alignment requirements of human cognitive performance studies? To answer these questions, this research contributes an exploration of establishing design patterns to reduce the cost of conducting human cognitive performance studies. The activities included in this exploration were a method for requirements gathering, a meta-study of recent experiments, and a design pattern evaluation all focused on the experimental design domain

A Novel Analysis of Performance Classification and Workload Prediction Using Electroencephalography (EEG) Frequency Data

Across the DOD each task an operator is presented with has some level of difficulty associated with it. This level of difficulty over the course of the task is also known as workload, where the operator is faced with varying levels of workload as he or she attempts to complete the task. The focus of the research presented in this thesis is to determine if those changes in workload can be predicted and to determine if individuals can be classified based on performance in order to prevent an increase in workload that would cause a decline in performance in a given task. Despite many efforts to predict workload and classify individuals with machine learning, the classification and predictive ability of Electroencephalography (EEG) frequency data has not been explored at the individual EEG Frequency band level. In a 711th HPW/RCHP Human Universal Measurement and Assessment Network (HUMAN) Lab study, 14 Subjects were asked to complete two tasks over 16 scenarios, while their physiological data, including EEG frequency data, was recorded to capture the physiological changes their body went through over the course of the experiment. The research presented in this thesis focuses on EEG frequency data, and its ability to predict task performance and changes in workload. Several machine learning techniques are explored in this thesis before a final technique was chosen. This thesis contributes research to the medical and machine learning fields regarding the classification and workload prediction efficacy of EEG frequency data. Specifically, it presents a novel investigation of five EEG frequencies and their individual abilities to predict task performance and workload. It was discovered that using the Gamma EEG frequency and all EEG frequencies combined to predict task performance resulted in average classification accuracies of greater than 90%

Attentional Narrowing: Triggering, Detecting and Overcoming a Threat to Safety.

In complex safety-critical domains, such as aviation or medicine, considerable multitasking requirements and attentional demands are imposed on operators who may, during off-nominal events, also experience high levels of anxiety. High task load and anxiety can trigger attentional narrowing – an involuntary reduction in the range of cues that can be utilized by an operator. As evidenced by numerous accidents, attentional narrowing is a highly undesirable and potentially dangerous state as it hampers information gathering, reasoning, and problem solving. However, because the problem is difficult to reproduce in controlled environments, little is known about its triggers, markers and possible countermeasures. Therefore, the goals of this dissertation were to (1) identify reliable triggers of attentional narrowing in controlled laboratory settings, (2) identify real-time markers of attentional narrowing that can also distinguish that phenomenon from focused attention – another state of reduced attentional field that, contrary to attentional narrowing, is deliberate and often desirable, (3) develop and test display designs that help overcome the narrowing of the attentional field. Based on a series of experiments in the context of a visual search task and a multi-tasking environment, novel unsolvable problems were identified as the most reliable trigger of attentional narrowing. Eye tracking was used successfully to detect and trace the phenomenon. Specifically, three eye tracking metrics emerged as promising markers of attentional narrowing: (1) the percentage of fixations, (2) dwell duration and (3) fixation duration in the display area where the novel problem was presented. These metrics were used to develop an algorithm capable of detecting attentional narrowing in real time and distinguishing it from focused attention. A command display (as opposed to status) was shown to support participants in broadening their attentional field and improving their time sharing performance. This dissertation contributes to the knowledge base in attentional narrowing and, more generally, attention management. A novel eye tracking based technique for detecting the attentional state and a promising countermeasure to the problem were developed. Overall, the findings from this research contribute to improved safety and performance in a range of complex high-risk domains.PHDIndustrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/135773/1/jprinet_1.pd

Estimation de l'état fonctionnel de l'opérateur

L’estimation de l’état fonctionnel de l’opérateur (c’est-à-dire le patron multidimensionnel de conditions physiologiques et comportementales qui régule les performances) a un grand potentiel pour l’augmentation de la sécurité et de la fiabilité de systèmes critiques. L’apprentissage automatique, qui a connu des avancées importantes au cours des dernières années, est une avenue à explorer pour effectuer cette estimation. Une problématique dans l’utilisation de ces techniques est la formalisation de l’OFS en une mesure objective permettant de fournir un signal d’entraînement à l’apprentissage automatique. Ce mémoire présente une mesure, la performance dynamique décontextualisée, permettant d’utiliser ces techniques pour estimer l’état fonctionnel de plusieurs participants, pour plusieurs tâches expérimentales différentes. Cet ouvrage explore également les performances obtenues par plusieurs techniques d’apprentissage automatique dans divers contextes. Entre autres, la généralisation des modèles entraînés à de nouveaux participants ou de nouvelles tâches expérimentales et l’utilisation du contexte expérimental sont étudiées.The assessment of an operator’s functional state (i.e., the multidimensional pattern of human psycho-physiological conditions that mediates performance) has great potential for increasing safety and reliability of critical systems. Machine learning, which has had success in recent years, is a technique which should be investigated for this task. An open question in the use of machine learning algorithms for the assessment of the operator’s functional state is the formalization of the operator’s state in an objective measure that can provide a training signal for the algorithms. This Master’s thesis introduces the decontextualized dynamic performance, a measure which enables the use of machine learning for many experimental tasks and many participants simultaneously.This work also explores the performances obtained by machine learning techniques in some contexts. The generalization of the trained models to new participants, or new tasks as well as the utilization of the training context is investigated

Eye Tracking: A Promising Means of Tracing, Explaining, and Preventing the Effects of Display Clutter in Real Time.

Display clutter is a widely-acknowledged but ill-defined problem that affects operators in complex, data-rich domains, such as medicine and aviation. Largely regarded a function of data density and display organization, clutter has been shown to degrade performance on a range of tasks, most notably visual search and noticing. Clutter effects may be exacerbated by stress, a major performance-shaping factor in the above domains. The goal of this dissertation was to develop an eye tracking-based approach for tracing and preventing the effects of clutter and stress on attention allocation and information acquisition. The research involved three stages: 1) identify the most diagnostic eye tracking metrics for capturing and explaining the effects of clutter and stress on performance, 2) determine which eye tracking metrics can detect the effects of clutter early on, in real time, and form the basis for models of clutter effects, and 3) evaluate the effectiveness of real-time display adjustments for preventing performance decrements. This research was carried out in several contexts, including emergency department (ED) electronic medical records (EMRs). First, three experiments were conducted in different application domains, including the ED, to establish the relationship between clutter, stress, attention, and performance during visual search and noticing tasks. Clutter resulted in performance decrements on both tasks. The underlying changes in attention allocation were captured by several eye tracking metrics, some of which were able to differentiate between the effects of data density and organization. A fourth experiment calculated the most promising eye tracking metrics in real time and used them as input to logistic regression models of response time. Long response time due to poor organization could be modeled most accurately. Finally, a fifth experiment presented ED physicians with real-time adaptations (highlighting and shortcut panel) to their EMR while they reviewed patient records to perform diagnoses. Both adjustments led to better performance and were viewed favorably by physicians. Overall, this research adds to the knowledge base on clutter and visual attention, supports the further development of eye tracking as a basis for real-time processing, and contributes to improved safety in various domains by supporting timely and accurate information acquisition.PhDIndustrial and Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/113627/1/nadmarie_1.pd

Diagnosticité des mesures physiologiques périphériques de la charge mentale

La charge mentale est un concept très utile dans les domaines dont l’objet d’étude et d’analyse est le travail et la performance humaine. Typiquement, la charge mentale est mesurée à l’aide des mesures subjectives (c.-à-d. des questionnaires) ou par des mesures comportementales (c.-à-d. les actions des individus). La charge mentale peut également être mesurée à l’aide de l’activité physiologique périphérique (p.ex. l’activité cardiaque). Il est cependant difficile de déterminer la source de la charge mentale à l’aide des mesures physiologiques périphérique. En effet, les sous-dimensions de la charge mentale, comme l’exigence, l’effort, le stress et la fatigue, provoquent souvent des manifestations physiologiques similaires. En plus de cette problématique, les mesures physiologiques de la charge mentale sont trop souvent étudiées dans des contextes de laboratoire et trop rarement étudiées dans des milieux de travail réels. Il est donc crucial d’investiguer leur potentiel dans des contextes réels. Cette thèse vise donc à investiguer le potentiel diagnostique (le potentiel à déterminer la source) des mesures physiologiques périphériques de la charge mentale. Pour y arriver, une méthode combinant l’approches cognitive traditionnelle et l’apprentissage automatique est utilisée. La thèse rapporte les résultats de deux expériences : une première menée en contexte de laboratoire et une seconde menée dans une simulation de commandement et contrôle reproduisant un milieu de travail réel. Les résultats montrent que les mesures physiologiques périphériques peuvent prédire, avec une bonne précision, la sousdimension qui est à l’origine de la charge mentale en contexte de tâche simple. Bien que moins précise, il reste possible de faire cette prédiction dans des contextes de tâche réelle. Dans l’ensemble, cette thèse apporte plusieurs contributions essentielles afin de rendre possible les mesures physiologiques périphériques de la charge mentale dans les milieux de travail réels.Mental workload stands out as a key concept as soon as human work and human performance is discussed. Mental workload is often measured using subjective questionnaires or behavioral cues. Peripheral physiological measures (e.g. heart rate) can also be used to measure workload. However, it is particularly difficult to determine the source of workload using peripheral physiological measures. Sub-divisions of mental workload, such as task load, mental effort, stress and fatigue, often trigger similar physiological reactions, blurring the diagnostic potential of physiological measures. Furthermore, physiological measures are too often investigated in laboratory settings, making it hazardous to determine their performance in real world settings. This thesis aims at investigating the diagnostic potential of peripheral physiological measures. A mixed methodology, combining traditional cognitive approach as well as machine learning techniques, is used. This thesis presents results of both a laboratory setting experimental as well as an ecological command and control simulation. Results show that peripheral measures can be used to predict, with high accuracy, the source of workload in laboratory settings. While not as accurate, results also show that it is possible to perform a diagnostic measure of workload in an ecological work simulation. This thesis contribute to improve the potential of peripheral physiological measures in real work settings