Queueing Network Modeling of Human Performance and Mental Workload in Perceptual-Motor Tasks.

Integrated with the mathematical modeling approaches, this thesis uses Queuing Network-Model Human Processors (QN-MHP) as a simulation platform to quantify human performance and mental workload in four representative perceptual-motor tasks with both theoretical and practical importance: discrete perceptual-motor tasks (transcription typing and psychological refractory period) and continuous perceptual-motor tasks (visual-manual tracking and vehicle steering with secondary tasks). The properties of queuing networks (queuing/waiting in processing information, serial and parallel information processing capability, overall mathematical structure, and entity-based network arrangement) allow QN-MHP to quantify several important aspects of the perceptual-motor tasks and unify them into one cognitive architecture. In modeling the discrete perceptual-motor task in a single task situation (transcription typing), QN-MHP quantifies and unifies 32 transcription typing phenomena involving many aspects of human performance--interkey time, typing units and spans, typing errors, concurrent task performance, eye movements, and skill effects, providing an alternative way to model this basic and common activities in human-machine interaction. In quantifying the discrete perceptual-motor task in a dual-task situation (psychological refractory period), the queuing network model is able to account for various experimental findings in PRP including all of these major counterexamples of existing models with less or equal number of free parameters and no need to use task-specific lock/unlock assumptions, demonstrating its unique advantages in modeling discrete dual-task performance. In modeling the human performance and mental workload in the continuous perceptual-motor tasks (visual-manual tracking and vehicle steering), QN-MHP is used as a simulation platform and a set of equations is developed to establish the quantitative relationships between queuing networks (e.g., subnetwork s utilization and arrival rate) and P300 amplitude measured by ERP techniques and subjective mental workload measured by NASA-TLX, predicting and visualizing mental workload in real-time. Moreover, this thesis also applies QN-MHP into the design of an adaptive workload management system in vehicles and integrates QN-MHP with scheduling methods to devise multimodal in-vehicle systems. Further development of the cognitive architecture in theory and practice is also discussed.Ph.D.Industrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/55678/2/changxuw_1.pd

Queuing Network Modeling of Human Multitask Performance and its Application to Usability Testing of In-Vehicle Infotainment Systems.

Human performance of a primary continuous task (e.g., steering a vehicle) and a secondary discrete task (e.g., tuning radio stations) simultaneously is a common scenario in many domains. It is of great importance to have a good understanding of the mechanisms of human multitasking behavior in order to design the task environments and user interfaces (UIs) that facilitate human performance and minimize potential safety hazards. In this dissertation I investigated and modeled human multitask performance with a vehicle-steering task and several typical in-vehicle secondary tasks. Two experiments were conducted to investigate how various display designs and control modules affect the driver's eye glance behavior and performance. A computational model based on the cognitive architecture of Queuing Network-Model Human Processor (QN-MHP) was built to account for the experiment findings. In contrast to most existing studies that focus on visual search in single task situations, this dissertation employed experimental work that investigates visual search in multitask situations. A modeling mechanism for flexible task activation (rather than strict serial activations) was developed to allow the activation of a task component to be based on the completion status of other task components. A task switching scheme was built to model the time-sharing nature of multitasking. These extensions offer new theoretical insights into visual search in multitask situations and enable the model to simulate parallel processing both within one task and among multiple tasks. The validation results show that the model could account for the observed performance differences from the empirical data. Based on this model, a computer-aided engineering toolkit was developed that allows the UI designers to make quantitative prediction of the usability of design concepts and prototypes. Scientifically, the results of this dissertation research offer additional insights into the mechanisms of human multitask performance. From the engineering application and practical value perspective, the new modeling mechanism and the new toolkit have advantages over the traditional usability testing methods with human subjects by enabling the UI designers to explore a larger design space and address usability issues at the early design stages with lower cost both in time and manpower.PHDIndustrial and Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/113590/1/fredfeng_1.pd

Pedestrian Dynamics: Modeling and Analyzing Cognitive Processes and Traffic Flows to Evaluate Facility Service Level

Walking is the oldest and foremost mode of transportation through history and the prevalence of walking has increased. Effective pedestrian model is crucial to evaluate pedestrian facility service level and to enhance pedestrian safety, performance, and satisfaction. The objectives of this study were to: (1) validate the efficacy of utilizing queueing network model, which predicts cognitive information processing time and task performance; (2) develop a generalized queueing network based cognitive information processing model that can be utilized and applied to construct pedestrian cognitive structure and estimate the reaction time with the first moment of service time distribution; (3) investigate pedestrian behavior through naturalistic and experimental observations to analyze the effects of environment settings and psychological factors in pedestrians; and (4) develop pedestrian level of service (LOS) metrics that are quick and practical to identify improvement points in pedestrian facility design. Two empirical and two analytical studies were conducted to address the research objectives. The first study investigated the efficacy of utilizing queueing network in modeling and predicting the cognitive information processing time. Motion capture system was utilized to collect detailed pedestrian movement. The predicted reaction time using queueing network was compared with the results from the empirical study to validate the performance of the model. No significant difference between model and empirical results was found with respect to mean reaction time. The second study endeavored to develop a generalized queueing network system so the task can be modeled with the approximated queueing network and its first moment of any service time distribution. There was no significant difference between empirical study results and the proposed model with respect to mean reaction time. Third study investigated methods to quantify pedestrian traffic behavior, and analyze physical and cognitive behavior from the real-world observation and field experiment. Footage from indoor and outdoor corridor was used to quantify pedestrian behavior. Effects of environmental setting and/or psychological factor on travel performance were tested. Finally, adhoc and tailor-made LOS metrics were presented for simple realistic service level assessments. The proposed methodologies were composed of space revision LOS, delay-based LOS, preferred walking speed-based LOS, and ‘blocking probability’

Queueing Network Modeling of Human Performance in Complex Cognitive Multi-task Scenarios.

As the complexity of human-machine systems grows rapidly, there is an increasing need for human factors theories and computational methods that can quantitatively model and simulate human performance and mental workload in complex multi-task scenarios. In response to this need, I have developed and evaluated an integrated cognitive architecture named QN-ACTR, which integrates two previously isolated but complementary cognitive architectures – Queueing Network (QN) and Adaptive Control of Thought-Rational (ACT-R). Combining their advantages and overcoming the limitations of each method, QN-ACTR possesses the benefits of modeling a wider range of tasks including multi-tasks with complex cognitive activities that existing methods have difficulty to model. These benefits have been evaluated and demonstrated by comparing model results with human results in the simulation of multi-task scenarios including skilled transcription typing and reading comprehension (human-computer interaction), medical decision making with concurrent tasks (healthcare), and driving with a secondary speech comprehension task (transportation), all of which contain important and practical human factors issues. QN-ACTR models produced performance and mental workload results similar to the human results. To support industrial applications of QN-ACTR, I have also developed the usability features of QN-ACTR to facilitate the use of this cognitive engineering tool by industrial and human factors engineers. Future research can apply QN-ACTR – which is a generic computational modeling theory and method – to other domains with important human factors issues.PHDIndustrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/102477/1/shicao_1.pd

The Virtual Driver: Integrating Physical and Cognitive Human Models to Simulate Driving with a Secondary In-Vehicle Task.

Models of human behavior provide insight into people’s choices and actions and form the basis of engineering tools for predicting performance and improving interface design. Most human models are either cognitive, focusing on the information processing underlying the decisions made when performing a task, or physical, representing postures and motions used to perform the task. In general, cognitive models contain a highly simplified representation of the physical aspects of a task and are best suited for analysis of tasks with only minor motor components. Physical models require a person experienced with the task and the software to enter detailed information about how and when movements should be made, a process that can be costly, time consuming, and inaccurate. Many tasks have both cognitive and physical components, which may interact in ways that could not be predicted using a cognitive or physical model alone. This research proposes a solution by combining a cognitive model, the Queuing Network – Model Human Processor, and a physical model, the Human Motion Simulation (HUMOSIM) Framework, to produce an integrated cognitive-physical human model that makes it possible to study complex human-machine interactions. The physical task environment is defined using the HUMOSIM Framework, which communicates relevant information such as movement times and difficulty to the QN-MHP. Action choice and movement sequencing are performed in the QN-MHP. The integrated model’s more natural movements, generated by motor commands from the QN-MHP, and more realistic cognitive decisions, made using physical information from the Framework, make it useful for evaluating different designs for tasks, spaces, systems, and jobs. The Virtual Driver is the application of the integrated model to driving with an in-vehicle task. A driving simulator experiment was used to tune and evaluate the integrated model. Increasing the visual and physical difficulty of the in-vehicle task affected the resource-sharing strategies drivers used and resulted in deterioration in driving and in-vehicle task performance, especially for shorter drivers. The Virtual Driver replicates basic driving, in-vehicle task, and resource-sharing behaviors and provides a new way to study driver distraction. The model has applicability to interface design and predictions about staffing requirements and performance.Ph.D.Biomedical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/75847/1/hjaf_1.pd

Modeling Dual-Task Concurrency and Effort in QN-ACTR and IMPRINT.

Computational cognitive models have wide ranging applications from reducing the time and cost of task and interface analyses to the discovery of new human cognitive phenomena. We investigate the use and limitations of IMPRINT, a task network simulation tool, and develop an extension to improve the modeling of task component execution limits in multi-task performance under high workload. The extension is implemented as a Soar agent that moderates task execution akin to executive processes in EPIC. We show that an IMPRINT model of a UAV operation task with the extension exhibits qualitatively distinct workload management strategies also observed in human performance of the same task. Next, we develop QN-ACTR models of a concurrent addition and targeting task and collect empirical data of human performance on the tasks to validate the models' predictions of execution time and a time sharing concurrency metric. We also use the empirical data to validate an IMPRINT model of the addition and targeting tasks. Both QN-ACTR and IMPRINT models capture the primary effects of variable task difficulty parameters on execution time and concurrency. Model inaccuracy at the subtask level provides evidence for the use of visual spatial memory during complex addition. In a second experiment with similar tasks, we introduce an incentive to examine the effects of effort on execution time and concurrency in dual task performance. Incentive induced effort is found to increase performance on the rewarded dimension without an increase in the time sharing concurrency metric, suggesting that the performance improvements are not derived from an increase in task scheduling efficiency or resource sharing but from the same improvements found in single task conditions. The QN-ACTR task models are modified to account for the increased effort by adjusting base level parameters and are validated with the empirical data.PhDIndustrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/102330/1/cjbest_1.pd

Anytime Cognition: An information agent for emergency response

Planning under pressure in time-constrained environments while relying on uncertain information is a challenging task. This is particularly true for planning the response during an ongoing disaster in a urban area, be that a natural one, or a deliberate attack on the civilian population. As the various activities pertaining to the emergency response need to be coordinated in response to multiple reports from the disaster site, a user finds itself cognitively overloaded. To address this issue, we designed the Anytime Cognition (ANTICO) concept to assist human users working in time-constrained environments by maintaining a manageable level of cognitive workload over time. Based on the ANTICO concept, we develop an agent framework for proactively managing a user’s changing information requirements by integrating information management techniques with probabilistic plan recognition. In this paper, we describe a prototype emergency response application in the context of a subset of the attacks devised by the American Department of Homeland Security

Eye-Tracking Metrics Predict Perceived Workload in Robotic Surgical Skills Training

Objective: The aim of this study is to assess the relationship between eye-tracking measures and perceived workload in robotic surgical tasks. Background: Robotic techniques provide improved dexterity, stereoscopic vision, and ergonomic control system over laparoscopic surgery, but the complexity of the interfaces and operations may pose new challenges to surgeons and compromise patient safety. Limited studies have objectively quantified workload and its impact on performance in robotic surgery. Although not yet implemented in robotic surgery, minimally intrusive and continuous eye-tracking metrics have been shown to be sensitive to changes in workload in other domains. Methods: Eight surgical trainees participated in 15 robotic skills simulation sessions. In each session, participants performed up to 12 simulated exercises. Correlation and mixed-effects analyses were conducted to explore the relationships between eye-tracking metrics and perceived workload. Machine learning classifiers were used to determine the sensitivity of differentiating between low and high workload with eye-tracking features. Results: Gaze entropy increased as perceived workload increased, with a correlation of .51. Pupil diameter and gaze entropy distinguished differences in workload between task difficulty levels, and both metrics increased as task level difficulty increased. The classification model using eye-tracking features achieved an accuracy of 84.7% in predicting workload levels. Conclusion: Eye-tracking measures can detect perceived workload during robotic tasks. They can potentially be used to identify task contributors to high workload and provide measures for robotic surgery training. Application: Workload assessment can be used for real-time monitoring of workload in robotic surgical training and provide assessments for performance and learning

Modelling Level 1 Situation Awareness in Driving: A Cognitive Architecture Approach

The goal of this research is to computationally model and simulate the collective drivers’ Level I Situation Awareness (SA). I developed a computational model in a cognitive architecture that can interact with a driving simulator to infer quantitative predictions of drivers’ SA. I demonstrate theoretical application of modelling and predicting SA from the lens of human cognition utilizing the Queueing Network-Adaptive Control of Thought Rational (QN-ACT-R) framework as a foundation. I integrated a dynamic visual sampling model (SEEV) with QN-ACT-R to create QN-ACT-R-SA which simulates realistic attention allocation patterns of human drivers at SA Level 1 (i.e. perception of critical elements). QN-ACT-R-SA also incorporates a driver model that can simulate human driving behaviors by interacting with a driving simulator. Three validation studies (Study I, II and III) were conducted to determine whether Level 1 SA results produced with the QN-ACT-R-SA model correspond to empirical data collected from human drivers for the same tasks. In Study I, QN-ACT-R-SA model was validated against probe-based SA measures and in Study II, the model was validated against a hazard perception-based SA measure. In Study III, model’s predictive power was assessed by comparing model results to a previously conducted empirical experiment. In Study I, two types of probe-based SA measures were used: within-task queries using Situation Awareness Global Assessment Technique (SAGAT), and post-experiment questions. A comparative assessment demonstrated that QN-ACT-R-SA could reasonably simulate drivers’ Level 1 SA for two driving conditions: easy (with few vehicles and signboards) and complex (with dense traffic and signboards). QN-ACT-R-SA fit for human SAGAT scores resulted in mean absolute percentage error (MAPE) of 5.02%, and the root mean square error (RMSE) of 3.47. Model fit for post-experiment human SA results were MAPE of 6.73%, and RMSE of 6.13. The RMSE of 3.47 for SAGAT responses indicate a small error difference between the average human and modelling results since the average SAGAT scores (measured on a scale of 0-100) for the easy and complex driving condition is around 71.9 (SD: 21.1). Similarly, the RMSE of 6.13 for post-experiment SA questionnaire also indicates a small error difference since the average post-experiment SA questionnaire score (on a scale of 0-100) for the easy and complex driving condition is around 73.8 (SD: 16.2). In Study II, Brake Perception Response Time (BPRT) was used as a hazard perception test to further assess the model’s ability to simulate drivers’ SA at Level 1. An empirical study was designed mainly for model validation purposes. In the trials runs, the participants encountered two major types of hazards: on-road hazards in the forward view of the driver and roadside hazard which originated from the driver’s periphery. The two contrasting conditions were selected to explore the difference in driver’s BPRT. The results demonstrated that BPRT was significantly shorter for on-road hazards as compared to roadside hazards. The overall model fitness for empirical BPRT results indicated an MAPE of 9.4 % and the RMSE of 0.13 seconds. The RMSE value in Study II indicates a small error difference between the average human and modelling results since the average BPRT for the two on-road and roadside hazard conditions is around 1.49 seconds (SD: 0.54). Study III involved extending the same modelling approach towards assessing the predictive power of QN-ACT-R-SA. The empirical data was taken from a previously conducted research study that had examined the effects of Adaptive Cruise Control (ACC) and cellphone use on drivers’ SA using SAGAT tests. QN-ACT-R-SA fit for predicting the effects of ACC and cellphone use on drivers’ Level 1 SA resulted in a MAPE of 5.6%, and the RMSE of 4.9. The RMSE of 4.9 for SAGAT responses indicates a small error difference between the average human and modelling results since the average SAGAT scores for the different driving conditions in Study III is around 72 (SD: 4.76). Both absolute (MAPE) and relative (RMSE) measures of goodness-of-fit confirm models efficacy in reasonably simulating human SA across the three studies. The MAPE value of less than 10% across the three studies show that the model’s deviation from the empirical results in terms of percentage error is relatively small. The graphical analysis of the average model versus average human plots further indicate that the model was able to successfully map the changes in SA scores across the different experimental conditions tested in the three studies. In summary, this research presents: 1) a model of collective drivers’ Level 1 SA that is grounded in cognitive and perceptual mechanisms of human information processing; 2) a real-time programmable implementation of the model as a simulation software; 3) validation of the model using empirical results drawn from established SA measures; and 4) new ideas towards modelling Level 2/3 SA and improving the existing modelling paradigm

Improving Human-Machine Interaction

This thesis studies human and machine interaction. For better interaction between humans and machines, this thesis aims to address three issues that remain unanswered in literature. Three objectives are proposed in this thesis to address the three issues, and the objectives are: (i) identification of the core capabilities of a Human Assistance System (HAS) and study of implementation strategy of the core capabilities; (ii) development of a framework for improving the accuracy of human mind state inference; (iii) study of the effect of representation of the machine’s state (which is represented in a “natural” way) on the user’s actions. By a natural way, it is meant a way that contains emotions known to be always present in humans (or human emotions in short). The study includes theoretical development, experimentation, and prototype implementation. This thesis has concluded: (1) the core capabilities to be addressed in designing a HAS are transparency, communication, rationale, cognition and task-sharing and they can be implemented with the existing technologies including fuzzy logics, Petri Net and ACT-R (Adaptive Control of Thought-Rational); (2) expert opinion elicitation technique is a promising method to construct a more general framework for integrating various algorithms on human state inference; (3) there is a significant effect of the representation of the machine’s state on the user’s actions. The main contributions of this thesis are: (1) provision of a case study for the proof-of-concept of HAS in the area of Computer Aided Design (CAD); (2) provision of an integrated framework for fatigue inference for improved accuracy, being readily generalized to inference of other mind states; (3) generation of a new knowledge regarding the effect of the natural representation of a machine’s states on the user’s actions. These contributions are significant in human-machine science and technology. The first contribution may lead to the development of a new generation CAD system in the near future. The second contribution provides a much powerful technology for human mind inference, which is a key capability in HAS, and the third contribution enriches the science of human-machine interaction and will give impact to the field of Artificial Intelligence (AI) as well. The application of the result of this thesis is rehabilitation, machine learning, etc