Optimizing The Design Of Multimodal User Interfaces

Due to a current lack of principle-driven multimodal user interface design guidelines, designers may encounter difficulties when choosing the most appropriate display modality for given users or specific tasks (e.g., verbal versus spatial tasks). The development of multimodal display guidelines from both a user and task domain perspective is thus critical to the achievement of successful human-system interaction. Specifically, there is a need to determine how to design task information presentation (e.g., via which modalities) to capitalize on an individual operator\u27s information processing capabilities and the inherent efficiencies associated with redundant sensory information, thereby alleviating information overload. The present effort addresses this issue by proposing a theoretical framework (Architecture for Multi-Modal Optimization, AMMO) from which multimodal display design guidelines and adaptive automation strategies may be derived. The foundation of the proposed framework is based on extending, at a functional working memory (WM) level, existing information processing theories and models with the latest findings in cognitive psychology, neuroscience, and other allied sciences. The utility of AMMO lies in its ability to provide designers with strategies for directing system design, as well as dynamic adaptation strategies (i.e., multimodal mitigation strategies) in support of real-time operations. In an effort to validate specific components of AMMO, a subset of AMMO-derived multimodal design guidelines was evaluated with a simulated weapons control system multitasking environment. The results of this study demonstrated significant performance improvements in user response time and accuracy when multimodal display cues were used (i.e., auditory and tactile, individually and in combination) to augment the visual display of information, thereby distributing human information processing resources across multiple sensory and WM resources. These results provide initial empirical support for validation of the overall AMMO model and a sub-set of the principle-driven multimodal design guidelines derived from it. The empirically-validated multimodal design guidelines may be applicable to a wide range of information-intensive computer-based multitasking environments

Virtual Reality Adaptation Using Electrodermal Activity to Support the User Experience

Virtual reality is increasingly used for tasks such as work and education. Thus, rendering scenarios that do not interfere with such goals and deplete user experience are becoming progressively more relevant. We present a physiologically adaptive system that optimizes the virtual environment based on physiological arousal, i.e., electrodermal activity. We investigated the usability of the adaptive system in a simulated social virtual reality scenario. Participants completed an n-back task (primary) and a visual detection (secondary) task. Here, we adapted the visual complexity of the secondary task in the form of the number of non-player characters of the secondary task to accomplish the primary task. We show that an adaptive virtual reality can improve users' comfort by adapting to physiological arousal regarding the task complexity. Our findings suggest that physiologically adaptive virtual reality systems can improve users' experience in a wide range of scenarios

Gamification as a neuroergonomic approach to improving interpersonal situational awareness in cyber defense

In cyber threat situations, the establishment of a shared situational awareness as a basis for cyber defense decision-making results from adequate communication of a Recognized Cyber Picture (RCP). RCPs consist of actively selected information and have the goal of accurately presenting the severity and potential consequences of the situation. RCPs must be communicated between individuals, but also between organizations, and often from technical to non-/less technical personnel. The communication of RCPs is subject to many challenges that may affect the transfer of critical information between individuals. There are currently no common best practices for training communication for shared situational awareness among cyber defense personnel. The Orient, Locate, Bridge (OLB) model is a pedagogic tool to improve communication between individuals during a cyber threat situation. According to the model, an individual must apply meta-cognitive awareness (O), perspective taking (L), and communication skills (B) to successfully communicate the RCP. Gamification (applying game elements to non-game contexts) has shown promise as an approach to learning. We propose a novel OLB-based Gamification design to improve dyadic communication for shared situational awareness among (technical and non-technical) individuals during a cyber threat situation. The design includes the Gamification elements of narrative, scoring, feedback, and judgment of self. The proposed concept contributes to the educational development of cyber operators from both military and civilian organizations responsible for defending and securing digital infrastructure. This is achieved by combining the elements of a novel communication model with Gamification in a context in urgent need for educational input.publishedVersio

Explainable AI over the Internet of Things (IoT): Overview, State-of-the-Art and Future Directions

Explainable Artificial Intelligence (XAI) is transforming the field of Artificial Intelligence (AI) by enhancing the trust of end-users in machines. As the number of connected devices keeps on growing, the Internet of Things (IoT) market needs to be trustworthy for the end-users. However, existing literature still lacks a systematic and comprehensive survey work on the use of XAI for IoT. To bridge this lacking, in this paper, we address the XAI frameworks with a focus on their characteristics and support for IoT. We illustrate the widely-used XAI services for IoT applications, such as security enhancement, Internet of Medical Things (IoMT), Industrial IoT (IIoT), and Internet of City Things (IoCT). We also suggest the implementation choice of XAI models over IoT systems in these applications with appropriate examples and summarize the key inferences for future works. Moreover, we present the cutting-edge development in edge XAI structures and the support of sixth-generation (6G) communication services for IoT applications, along with key inferences. In a nutshell, this paper constitutes the first holistic compilation on the development of XAI-based frameworks tailored for the demands of future IoT use cases.Comment: 29 pages, 7 figures, 2 tables. IEEE Open Journal of the Communications Society (2022

Gamification as a neuroergonomic approach to improving interpersonal situational awareness in cyber defense

In cyber threat situations, the establishment of a shared situational awareness as a basis for cyber defense decision-making results from adequate communication of a Recognized Cyber Picture (RCP). RCPs consist of actively selected information and have the goal of accurately presenting the severity and potential consequences of the situation. RCPs must be communicated between individuals, but also between organizations, and often from technical to non−/less technical personnel. The communication of RCPs is subject to many challenges that may affect the transfer of critical information between individuals. There are currently no common best practices for training communication for shared situational awareness among cyber defense personnel. The Orient, Locate, Bridge (OLB) model is a pedagogic tool to improve communication between individuals during a cyber threat situation. According to the model, an individual must apply meta-cognitive awareness (O), perspective taking (L), and communication skills (B) to successfully communicate the RCP. Gamification (applying game elements to non-game contexts) has shown promise as an approach to learning. We propose a novel OLB-based Gamification design to improve dyadic communication for shared situational awareness among (technical and non-technical) individuals during a cyber threat situation. The design includes the Gamification elements of narrative, scoring, feedback, and judgment of self. The proposed concept contributes to the educational development of cyber operators from both military and civilian organizations responsible for defending and securing digital infrastructure. This is achieved by combining the elements of a novel communication model with Gamification in a context in urgent need for educational input

Unveiling AI Aversion: Understanding Antecedents and Task Complexity Effects

Artificial Intelligence (AI) has generated significant interest due to its potential to augment human intelligence. However, user attitudes towards AI are diverse, with some individuals embracing it enthusiastically while others harbor concerns and actively avoid its use. This two essays\u27 dissertation explores the reasons behind user aversion to AI. In the first essay, I develop a concise research model to explain users\u27 AI aversion based on the theory of effective use and the adaptive structuration theory. I then employ an online experiment to test my hypotheses empirically. The multigroup analysis by Structural Equation Modeling shows that users\u27 perceptions of human dissimilarity, AI bias, and social influence strongly drive AI aversion. Moreover, I find a significant difference between the simple and the complex task groups. This study reveals why users avert using AI by systematically examining the factors related to technology, user, task, and environment, thus making a significant contribution to the emerging field of AI aversion research. Next, while trust and distrust have been recognized as influential factors shaping users\u27 attitudes towards IT artifacts, their intricate relationship with task characteristics and their impact on AI aversion remains largely unexplored. In my second essay, I conduct an online randomized controlled experiment on Amazon Mechanical Turk to bridge this critical research gap. My comprehensive analytic approach, including structural equation modeling (SEM), ANOVA, and PROCESS conditional analysis, allowed me to shed light on the intricate web of factors influencing users\u27 AI aversion. I discovered that distrust and trust mediate between task complexity and AI aversion. Moreover, this study unveiled intriguing differences in these mediated relationships between subjective and objective task groups. Specifically, my findings demonstrate that, for objective tasks, task complexity can significantly increase aversion by reducing trust and significantly decrease aversion by reducing distrust. In contrast, for subjective tasks, task complexity only significantly increases aversion by enhancing distrust. By considering various task characteristics and recognizing trust and distrust as vital mediators, my research not only pushes the boundaries of the human-AI literature but also significantly contributes to the field of AI aversion

Holographic Generative Memory: Neurally Inspired One-Shot Learning with Memory Augmented Neural Networks

Humans quickly parse and categorize stimuli by combining perceptual information and previously learned knowledge. We are capable of learning new information quickly with only a few observations, and sometimes even a single observation. This one-shot learning (OSL) capability is still very difficult to realize in machine learning models. Novelty is commonly thought to be the primary driver for OSL. However, neuroscience literature shows that biological OSL mechanisms are guided by uncertainty, rather than novelty, motivating us to explore this idea for machine learning. In this work, we investigate OSL for neural networks using more robust compositional knowledge representations and a biologically inspired uncertainty mechanism to modulate the rate of learning. We introduce several new neural network models that combine Holographic Reduced Representation (HRR) and Variational Autoencoders. Extending these new models culminates in the Holographic Generative Memory (HGMEM) model. HGMEM is a novel unsupervised memory augmented neural network. It offers solutions to many of the practical drawbacks associated with HRRs while also providing storage, recall, and generation of latent compositional knowledge representations. Uncertainty is measured as a native part of HGMEM operation by applying trained probabilistic dropout to fully-connected layers. During training, the learning rate is modulated using these uncertainty measurements in a manner inspired by our motivating neuroscience mechanism for OSL. Model performance is demonstrated on several image datasets with experiments that reflect our theoretical approach

Attentional Narrowing: Triggering, Detecting and Overcoming a Threat to Safety.

In complex safety-critical domains, such as aviation or medicine, considerable multitasking requirements and attentional demands are imposed on operators who may, during off-nominal events, also experience high levels of anxiety. High task load and anxiety can trigger attentional narrowing – an involuntary reduction in the range of cues that can be utilized by an operator. As evidenced by numerous accidents, attentional narrowing is a highly undesirable and potentially dangerous state as it hampers information gathering, reasoning, and problem solving. However, because the problem is difficult to reproduce in controlled environments, little is known about its triggers, markers and possible countermeasures. Therefore, the goals of this dissertation were to (1) identify reliable triggers of attentional narrowing in controlled laboratory settings, (2) identify real-time markers of attentional narrowing that can also distinguish that phenomenon from focused attention – another state of reduced attentional field that, contrary to attentional narrowing, is deliberate and often desirable, (3) develop and test display designs that help overcome the narrowing of the attentional field. Based on a series of experiments in the context of a visual search task and a multi-tasking environment, novel unsolvable problems were identified as the most reliable trigger of attentional narrowing. Eye tracking was used successfully to detect and trace the phenomenon. Specifically, three eye tracking metrics emerged as promising markers of attentional narrowing: (1) the percentage of fixations, (2) dwell duration and (3) fixation duration in the display area where the novel problem was presented. These metrics were used to develop an algorithm capable of detecting attentional narrowing in real time and distinguishing it from focused attention. A command display (as opposed to status) was shown to support participants in broadening their attentional field and improving their time sharing performance. This dissertation contributes to the knowledge base in attentional narrowing and, more generally, attention management. A novel eye tracking based technique for detecting the attentional state and a promising countermeasure to the problem were developed. Overall, the findings from this research contribute to improved safety and performance in a range of complex high-risk domains.PHDIndustrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/135773/1/jprinet_1.pd

A Conceptual Framework to Support Digital Transformation in Manufacturing Using an Integrated Business Process Management Approach

Digital transformation is no longer a future trend, as it has become a necessity for businesses to grow and remain competitive in the market. The fourth industrial revolution, called Industry 4.0, is at the heart of this transformation, and is supporting organizations in achieving benefits that were unthinkable a few years ago. The impact of Industry 4.0 enabling technologies in the manufacturing sector is undeniable, and their correct use offers benefits such as improved productivity and asset performance, reduced inefficiencies, lower production and maintenance costs, while enhancing system agility and flexibility. However, organizations have found the move towards digital transformation extremely challenging for several reasons, including a lack of standardized implementation protocols, emphasis on the introduction of new technologies without assessing their role within the business, the compartmentalization of digital initiatives from the rest of the business, and the large-scale implementation of digitalization without a realistic view of return on investment. To instill confidence and reduce the anxiety surrounding Industry 4.0 implementation in the manufacturing sector, this paper presents a conceptual framework based on business process management (BPM). The framework is informed by a content-centric literature review of Industry 4.0 technologies, its design principles, and BPM method. This integrated framework incorporates the factors that are often overlooked during digital transformation and presents a structured methodology that can be employed by manufacturing organizations to facilitate their transition towards Industry 4.0