Computational Modeling and Experimental Research on Touchscreen Gestures, Audio/Speech Interaction, and Driving

As humans are exposed to rapidly evolving complex systems, there are growing needs for humans and systems to use multiple communication modalities such as auditory, vocal (or speech), gesture, or visual channels; thus, it is important to evaluate multimodal human-machine interactions in multitasking conditions so as to improve human performance and safety. However, traditional methods of evaluating human performance and safety rely on experimental settings using human subjects which require costly and time-consuming efforts to conduct. To minimize the limitations from the use of traditional usability tests, digital human models are often developed and used, and they also help us better understand underlying human mental processes to effectively improve safety and avoid mental overload. In this regard, I have combined computational cognitive modeling and experimental methods to study mental processes and identify differences in human performance/workload in various conditions, through this dissertation research. The computational cognitive models were implemented by extending the Queuing Network-Model Human Processor (QN-MHP) Architecture that enables simulation of human multi-task behaviors and multimodal interactions in human-machine systems. Three experiments were conducted to investigate human behaviors in multimodal and multitasking scenarios, combining the following three specific research aims that are to understand: (1) how humans use their finger movements to input information on touchscreen devices (i.e., touchscreen gestures), (2) how humans use auditory/vocal signals to interact with the machines (i.e., audio/speech interaction), and (3) how humans drive vehicles (i.e., driving controls). Future research applications of computational modeling and experimental research are also discussed. Scientifically, the results of this dissertation research make significant contributions to our better understanding of the nature of touchscreen gestures, audio/speech interaction, and driving controls in human-machine systems and whether they benefit or jeopardize human performance and safety in the multimodal and concurrent task environments. Moreover, in contrast to the previous models for multitasking scenarios mainly focusing on the visual processes, this study develops quantitative models of the combined effects of auditory, tactile, and visual factors on multitasking performance. From the practical impact perspective, the modeling work conducted in this research may help multimodal interface designers minimize the limitations of traditional usability tests and make quick design comparisons, less constrained by other time-consuming factors, such as developing prototypes and running human subjects. Furthermore, the research conducted in this dissertation may help identify which elements in the multimodal and multitasking scenarios increase workload and completion time, which can be used to reduce the number of accidents and injuries caused by distraction.PHDIndustrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/143903/1/heejinj_1.pd

RAPID PROTOTYPING Multipurpose Prototypes for Assessing User Interfaces in Pervasive Computing Systems

In pervasive computing systems, prototypes can communicate design ideas, support user testing, and predict skilled performance. Designers can use CogTool to build HTML storyboards as prototypes for these purposes

Experimental user interface design toolkit for interaction research (IDTR).

The research reported and discussed in this thesis represents a novel approach to User Interface evaluation and optimisation through cognitive modelling. This is achieved through the development and testing of a toolkit or platform titled Toolkit for Optimisation of Interface System Evolution (TOISE). The research is conducted in two main phases. In phase 1, the Adaptive Control of Thought Rational (ACT-R) cognitive architecture is used to design Simulated Users (SU) models. This allows models of user interaction to be tested on a specific User Interface (UI). In phase 2, an evolutionary algorithm is added and used to evolve and test an optimised solution to User Interface layout based on the original interface design. The thesis presents a technical background, followed by an overview of some applications in their respective fields. The core concepts behind TOISE are introduced through a discussion of the Adaptive Control of Thought “ Rational (ACT-R) architecture with a focus on the ACT-R models that are used to simulate users. The notion of adding a Genetic Algorithm optimiser is introduced and discussed in terms of the feasibility of using simulated users as the basis for automated evaluation to optimise usability. The design and implementation of TOISE is presented and discussed followed by a series of experiments that evaluate the TOISE system. While the research had to address and solve a large number of technical problems the resulting system does demonstrate potential as a platform for automated evaluation and optimisation of user interface layouts. The limitations of the system and the approach are discussed and further work is presented. It is concluded that the research is novel and shows considerable promise in terms of feasibility and potential for optimising layout for enhanced usability

Fundamental understanding and future guidance for handheld computers in the rail industry

Advances in mobile computing technology and software applications have led to an expansion in potential uses for handheld computers for various tasks. One strong application area is in maintenance and inspection. Network Rail has been progressively developing and applying handheld computers to field-based maintenance and inspection operations, with the aims of improving work productivity and quality, and personal and system safety. However, it is clear that these aims so far have been achieved with varying degrees of success. Handheld computer devices have the potential to enhance the procedure of performing the tasks in many different ways. However, the current handheld computers introduced to maintenance and inspection tasks in Network Rail have principally been designed as data entry tools and in most cases the primary objective is to reduce the amount of paper work and the associated costs and errors. This highlights the need for fundamental research into the ways in which handheld computer technologies should be specified, designed and implemented for effective use in a complex distributed environment such as the rail industry. The main purpose of this research was to study the applications of handheld computers in the rail industry and to generate a set of design principles for development of future systems within Network Rail. The findings of this research have contributed to the identification of human factors principles that need to be considered for design and implementation of successful handheld computer applications. A framework was also developed to summarise and organise information and functional requirements of maintenance workers. Investigating maintenance workers’ requirements through interviews and observations emphasised the importance of rail specific spatial information and the benefits of providing this knowledge to maintenance workers through a mobile computing device which is portable and easy to use. However, displaying rail specific spatial information on the small screen of a handheld computer introduces various HCI issues and challenges. These were addressed in part through a programme of experiments, and therefore the final section of this research focused on examining fundamental aspects of presenting rail specific spatial information on handheld computer screens. The main findings from different stages of this research have been collated into a set of recommendations for design and development of usable and useful applications for handheld computer devices in the rail industry