Modeling of Stimulus-Response Secondary Tasks with Different Modalities while Driving in a Computational Cognitive Architecture

This paper introduces a computational human performance model based upon the queueing network cognitive architecture to predict driver’s eye glances and workload for four stimulus-response secondary tasks (i.e., auditorymanual, auditory-speech, visual-manual, and visual-speech types) while driving. The model was evaluated with the empirical data from 24 subjects, and the percentage of eyes-off-road time and driver workload generated by the model were similar to the human subject data. Future studies aim to extend the types of voice announcements/commands to enable Human-Machine-Interface (HMI) evaluations with a wider range of usability test for in-vehicle infotainment system developments

Anticipatory Eye Movements in Interleaving Templates of Human Behavior

Performance modeling has been made easier by architectures which package psychological theory for reuse at useful levels of abstraction. CPM-GOMS uses templates of behavior to package at a task level (e.g., mouse move-click, typing) predictions of lower-level cognitive, perceptual, and motor resource use. CPM-GOMS also has a theory for interleaving resource use between templates. One example of interleaving is anticipatory eye movements. This paper describes the use of ACT-Stitch, a framework for translating CPM-GOMS templates and interleaving theory into ACT-R, to model anticipatory eye movements in skilled behavior. The anticipatory eye movements explain performance in a well-practiced perceptual/motor task, and the interleaving theory is supported with results from an eye-tracking experiment

An analysis of Space Shuttle countdown activities: Preliminaries to a computational model of the NASA Test Director

Before all systems are go just prior to the launch of a space shuttle, thousands of operations and tests have been performed to ensure that all shuttle and support subsystems are operational and ready for launch. These steps, which range from activating the orbiter's flight computers to removing the launch pad from the itinerary of the NASA tour buses, are carried out by launch team members at various locations and with highly specialized fields of expertise. The liability for coordinating these diverse activities rests with the NASA Test Director (NTD) at NASA-Kennedy. The behavior is being studied of the NTD with the goal of building a detailed computational model of that behavior; the results of that analysis to date are given. The NTD's performance is described in detail, as a team member who must coordinate a complex task through efficient audio communication, as well as an individual taking notes and consulting manuals. A model of the routine cognitive skill used by the NTD to follow the launch countdown procedure manual was implemented using the Soar cognitive architecture. Several examples are given of how such a model could aid in evaluating proposed computer support systems

Computational Modeling and Experimental Research on Touchscreen Gestures, Audio/Speech Interaction, and Driving

As humans are exposed to rapidly evolving complex systems, there are growing needs for humans and systems to use multiple communication modalities such as auditory, vocal (or speech), gesture, or visual channels; thus, it is important to evaluate multimodal human-machine interactions in multitasking conditions so as to improve human performance and safety. However, traditional methods of evaluating human performance and safety rely on experimental settings using human subjects which require costly and time-consuming efforts to conduct. To minimize the limitations from the use of traditional usability tests, digital human models are often developed and used, and they also help us better understand underlying human mental processes to effectively improve safety and avoid mental overload. In this regard, I have combined computational cognitive modeling and experimental methods to study mental processes and identify differences in human performance/workload in various conditions, through this dissertation research. The computational cognitive models were implemented by extending the Queuing Network-Model Human Processor (QN-MHP) Architecture that enables simulation of human multi-task behaviors and multimodal interactions in human-machine systems. Three experiments were conducted to investigate human behaviors in multimodal and multitasking scenarios, combining the following three specific research aims that are to understand: (1) how humans use their finger movements to input information on touchscreen devices (i.e., touchscreen gestures), (2) how humans use auditory/vocal signals to interact with the machines (i.e., audio/speech interaction), and (3) how humans drive vehicles (i.e., driving controls). Future research applications of computational modeling and experimental research are also discussed. Scientifically, the results of this dissertation research make significant contributions to our better understanding of the nature of touchscreen gestures, audio/speech interaction, and driving controls in human-machine systems and whether they benefit or jeopardize human performance and safety in the multimodal and concurrent task environments. Moreover, in contrast to the previous models for multitasking scenarios mainly focusing on the visual processes, this study develops quantitative models of the combined effects of auditory, tactile, and visual factors on multitasking performance. From the practical impact perspective, the modeling work conducted in this research may help multimodal interface designers minimize the limitations of traditional usability tests and make quick design comparisons, less constrained by other time-consuming factors, such as developing prototypes and running human subjects. Furthermore, the research conducted in this dissertation may help identify which elements in the multimodal and multitasking scenarios increase workload and completion time, which can be used to reduce the number of accidents and injuries caused by distraction.PHDIndustrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/143903/1/heejinj_1.pd

Task Analysis, Modeling, And Automatic Identification Of Elemental Tasks In Robot-Assisted Laparoscopic Surgery

Robotic microsurgery provides many advantages for surgical operations, including tremor filtration, an increase in dexterity, and smaller incisions. There is a growing need for a task analyses on robotic laparoscopic operations to understand better the tasks involved in robotic microsurgery cases. A few research groups have conducted task observations to help systems automatically identify surgeon skill based on task execution. Their gesture analyses, however, lacked depth and their class libraries were composed of ambiguous groupings of gestures that did not share contextual similarities. A Hierarchical Task Analysis was performed on a four-throw suturing task using a robotic microsurgical platform. Three skill levels were studied: attending surgeons, residents, and naïve participants. From this task analysis, a subtask library was created. The Hierarchical Task Analysis subtask library, a computer system was created that accurately identified surgeon subtasks based on surgeon hand gestures. An automatic classifier was trained on the subtasks identified during the Hierarchical Task Analysis of a four-throw suturing task and the motion signature recorded during task performance. Using principal component analysis and a J48 decision tree classifier, an average individual classification accuracy of 94.56% was achieved. This research lays the foundation for accurate and meaningful autonomous computer assistance in a surgical arena by creating a gesture library from a detailed Hierarchical Task Analysis. The results of this research will improve the surgeon-robot interface and enhance surgery performance. The classes used will eliminate human machine miscommunication by using an understandable and structured class library based on a Hierarchical Task Analysis. By enabling a robot to understand surgeon actions, intelligent contextual-based assistance could be provide to the surgeon by the robot. Limitations of this research included the small participant sample size used for this research which resulted in high subtask execution variability. Future work will include a larger participant population to address this limitation. Additionally, a Hidden Markov Model will be incorporated into the classification process to help increase the classification accuracy. Finally, a closer investigation of vestigial techniques will be conducted to study the effect of past learned laparoscopic techniques, which are no longer necessary in the robotic-assisted laparoscopic surgery arena

Modeling Complex High Level Interactions in the Process of Visual Mining

Visual Mining refers to the human analytical process that uses visual representations of raw data and makes suitable inferences. During this analytical process, users are engaged in complex cognitive activities such as decision making, problem solving, analytical reasoning and learning. Now a days, users typically use interactive visualization tools, which we call as visual mining support tools (VMSTs), to mediate their interactions with the information present in visual representations of raw data and also to support their complex cognitive activities when performing visual mining. VMSTs have two main components: visual representation and interaction. Even though, these two components are fundamental aspects of VMSTs, the research on visual representation has received the most attention. It is still unclear how to design interactions which can properly support users in performing complex cognitive activities during the visual mining process. Although some fundamental concepts and techniques regarding interaction design have been in place for a while, many established researchers are of the opinion that we do not yet have a generalized, principled, and systematic understanding of interaction components of these VMSTs, and how interactions should be analyzed, designed, and integrated to support complex cognitive activities. Many researchers have recommended that one way to address this problem is through appropriate characterization of interactions in the visual mining process. Models that provide classifications of interactions have indeed been proposed in the visualization research community. While these models are important contributions for the visualization research community, they often characterize interactions at lower levels of human information interaction and high level interactions are not well addressed. In addition, some of these models are not designed to model user activity; rather they are most applicable for representing a system’s response to user activity and not the user activity itself. In this thesis, we address this problem through characterization of the interaction space of visual mining at the appropriate level. Our main contribution in this research is the discovery of a small set of classification criteria which can comprehensively characterize the interaction space of visual mining involving interactions with VMSTs for performing complex cognitive activities. These complex cognitive activities are modeled through visual mining episodes, a coherent set of activities consisting of visual mining strategies (VMSs). Using the classification criteria, VMSs are simply described as combinations of different values of these criteria. By considering all combinations, we can comprehensively cover the interaction space of visual mining. Our VMS interaction space model is unique in identifying the activity tier, a granularity of interactions (high level) which supports performance of complex cognitive activities through interactions with visual information using VMSTs. As further demonstration of the utility of this VMS interaction space model, we describe the formulation of an inspection framework which can provide quantitative measures for the support provided by VMSTs for complex cognitive activities in visual mining. This inspection framework, which has enabled us to produce a new simpler evaluation method for VMSTs in comparison to existing evaluation methods, is based soundly on existing theories and models. Both the VMS interaction space model and the inspection framework present many interesting avenues for further research

Math in the Dark: Tools for Expressing Mathematical Content by Visually Impaired Students

Blind and visually impaired students are under-represented in the science, technology, engineering, and mathematics disciplines of higher education and the workforce. This is due primarily to the difficulties they encounter in trying to succeed in mathematics courses. While there are sufficient tools available to create Braille content, including the special Nemeth Braille used in the U.S. for mathematics constructs, there are very few tools to allow a blind or visually impaired student to create his/her own mathematical content in a manner that sighted individuals can use. The software tools that are available are isolated, do not interface well with other common software, and may be priced for institutional use instead of individual use. Instructors are unprepared or unable to interact with these students in a real-time manner. All of these factors combine to isolate the blind or visually impaired student in the study of mathematics. Nemeth Braille is a complete mathematical markup system in Braille, containing everything that is needed to produce quality math content at all levels of complexity. Blind and visually impaired students should not have to learn any additional markup languages in order to produce math content. This work addressed the needs of the individual blind or visually impaired student who must be able to produce mathematical content for course assignments, and who wishes to interact with peers and instructors on a real-time basis to share mathematical content. Two tools were created to facilitate mathematical interaction: a Nemeth Braille editor, and a real-time instant messenger chat capability that supports Nemeth Braille and MathML constructs. In the Visually Impaired view, the editor accepts Nemeth Braille input, displays the math expressions in a tree structure which will allow sub-expressions to be expanded or collapsed. The Braille constructs can be translated to MathML for display within MathType. Similarly, in the Sighted view, math constructs entered in MathType can be translated into Nemeth Braille. Mathematical content can then be shared between sighted and visually impaired users via the instant messenger chat capability. Using Math in the Dark software, blind and visually impaired students can work math problems fully in Nemeth Braille and can seamlessly convert their work into MathML for viewing by sighted instructors. The converted output has the quality of professionally produced math content. Blind and VI students can also communicate and share math constructs with a sighted partner via a real-time chat feature, with automatic translation in both directions, allowing VI students to obtain help in real-time from a sighted instructor or tutor. By eliminating the burden of translation, this software will help to remove the barriers faced by blind and VI students who wish to excel in the STEM fields of study

An empirical study of virtual reality menu interaction and design

This study focused on three different menu designs each with their own unique interactions and organizational structures to determine which design features would perform the best. Fifty-four participants completed 27 tasks using each of the three designs. The menus were analyzed based on task performance, accuracy, usability, intuitiveness, and user preference. Also, an analysis was conducted between two different menu organization styles: top-down menu organization (Method-TD) and bottom-up organization (Method-BU). There was no evidence that demographic factors had any effect on the overall results. By and large, the Stacked menu design received very positive results and feedback from all the participants. The Spatial design received average feedback with some participants preferring it while others struggled to use it and felt that it was too physically demanding. The worst performer was the Radial design that consistently ranked last and failed to pass usability and accuracy tests. A NGOMSL study was conducted to determine any differences in performance between a top-down menu organizational approach and a bottom-up approach or differences between the predicted task completion times and the reported times. The results of this study predicted that the Spatial design should have taken the least amount of time to perform, however, the experimental results showed that the Stacked design in fact out-performed the Spatial design’s task completion times. A potential explanation as to why the Stacked outperformed the Spatial is the increased physical demand of the Spatial design not anticipated with the NGOMSL analysis because of a design feature which caused a high level of cumbersomeness with the interactions. Overall, there were no statistical differences found between Method-TD and Method-BU, but a large difference found between the predicted times and observed times for Stacked, Radial, and Spatial. Participants overwhelmingly performed better than the predicted completion times for the Stacked design, but then did not complete the tasks by the predicted times for the Radial and Spatial. This study recommends the Stacked menu for VR environments and proposes further research into a Stacked-Spatial hybrid design to allow for the participant’s preferred design aspects of both designs to be used in a VR environment