Fuzzy Modelling of Human Psycho-Physiological State and Fuzzy Adaptive Control of Automation in Human-Machine Interface

This research aims at proposing a new modelling and control framework that monitors the human operators' psychophysiological state in the human-machine interface to prevent performance breakdown. This research started with the exploration of new psychophysiological state assessment approaches to the adaptive modelling and control method for predicting human task performance and balancing the engagement of the human operator and the automatic system. The results of this research may also be further applied in developing advanced control mechanisms, investigating the origins of human compromised performance and identifying or even remedying operators' breakdown in the early stages of operation, at least. A summary of the current human psychophysiological studies, previous human-machine interface simulation and existing biomarkers for human psychophysiological state assessment was provided for simulation experiment design of this research. The use of newly developed facial temperature biomarkers for assessing the human psychophysiological state and the task performance was investigated. The research continued by exploring the uncertainty of the human-machine interface system through the use of the complex fuzzy logic based offline modelling approach. A new type-2 fuzzy-based modelling approach was then proposed to assess the human operators' psychophysiological states in the real-time human-machine interface. This new modelling technique integrated state tracking and type-2 fuzzy sets for updating the rule base with a Bayesian process. Finally, this research included a new type-2 fuzzy logic-based control algorithm for balancing the human-machine interface systems via adjusting the engagement of the human operators according to their psychophysiological state and task performance. This innovative control approach combined the state estimation of the human operator with the type-2 fuzzy sets to maintain the balance between the task requirements (i.e. difficulty level) and the human operator feasible effort (i.e. psychophysiological states). In addition, the research revealed the impacts of multi-tasking and general fatigue on human operator's performance

An adaptive general type-2 fuzzy logic approach for psychophysiological state modeling in real-time human–machine interfaces

In this article, a new type-2 fuzzy-based modeling approach is proposed to assess human operators’ psychophysiological states for both safety and reliability of human–machine interface systems. Such a new modeling technique combines type-2 fuzzy sets with state tracking to update the rule base through a Bayesian process. These new configurations successfully lead to an adaptive, robust, and transparent computational framework that can be utilized to identify dynamic (i.e., real time) features without prior training. The proposed framework is validated on mental arithmetic cognitive real-time experiments with ten participants. It is found that the proposed framework outperforms other paradigms (i.e., an adaptive neuro-fuzzy inference system and an adaptive general type-2 fuzzy c-means modeling approach) in terms of disturbance rejection and learning capabilities. The proposed framework achieved the best performance compared to other models that have been presented in the related literature. Therefore, the new framework can be a promising development in human–machine interface systems. It can be further utilized to develop advanced control mechanisms, investigate the origins of human compromised task performance, and identify and remedy psychophysiological breakdown in the early stages

Psycho-Physiologically-Based Real Time Adaptive General Type 2 Fuzzy Modelling and Self-Organising Control of Operator's Performance Undertaking a Cognitive Task

—This paper presents a new modelling and control fuzzy-based framework validated with real-time experiments on human participants experiencing stress via mental arithmetic cognitive tasks identified through psycho-physiological markers. The ultimate aim of the modelling/control framework is to prevent performance breakdown in human-computer interactive systems with a special focus on human performance. Two designed modelling/control experiments which consist of carrying-out arithmetic operations of varying difficulty levels were performed by 10 participants (operators) in the study. With this new technique, modelling is achieved through a new adaptive, self-organizing and interpretable modelling framework based on General Type-2 Fuzzy sets. This framework is able to learn in real-time through the implementation of a re-structured performance-learning algorithm that identifies important features in the data without the need for prior training. The information learnt by the model is later exploited via an Energy Model Based Controller that infers adequate control actions by changing the difficulty level of the arithmetic operations in the human-computer-interaction system; these actions being based on the most current psycho-physiological state of the subject under study. The real-time implementation of the proposed modelling and control configurations for the human-machine-interaction under study shows superior performance as compared to other forms of modelling and control, with minimal intervention in terms of model re-training or parameter re-tuning to deal with uncertainties, disturbances and inter/intra-subject parameter variability

Enabling Single-Pilot Operations technological and operative scenarios: a state-of-the-art review with possible cues

Both financial and operational reasons have been given emphasis to the implementation of Single-Pilot Operations in commercial aviation. SPO will involve replacing the first officer with integrated cockpit assistants and support ground stations. This review aims to provide an overview of SPO through a classification of the specific areas of interest. Enabling SPO will require designers to re-modulate the human-automation interface according to the new allocation of functions in the flight deck. However, while technological issues are expected to be overcome in the next future, major attention should be paid on the human factor side

Applications of Affective Computing in Human-Robot Interaction: state-of-art and challenges for manufacturing

The introduction of collaborative robots aims to make production more flexible, promoting a greater interaction between humans and robots also from physical point of view. However, working closely with a robot may lead to the creation of stressful situations for the operator, which can negatively affect task performance. In Human-Robot Interaction (HRI), robots are expected to be socially intelligent, i.e., capable of understanding and reacting accordingly to human social and affective clues. This ability can be exploited implementing affective computing, which concerns the development of systems able to recognize, interpret, process, and simulate human affects. Social intelligence is essential for robots to establish a natural interaction with people in several contexts, including the manufacturing sector with the emergence of Industry 5.0. In order to take full advantage of the human-robot collaboration, the robotic system should be able to perceive the psycho-emotional and mental state of the operator through different sensing modalities (e.g., facial expressions, body language, voice, or physiological signals) and to adapt its behaviour accordingly. The development of socially intelligent collaborative robots in the manufacturing sector can lead to a symbiotic human-robot collaboration, arising several research challenges that still need to be addressed. The goals of this paper are the following: (i) providing an overview of affective computing implementation in HRI; (ii) analyzing the state-of-art on this topic in different application contexts (e.g., healthcare, service applications, and manufacturing); (iii) highlighting research challenges for the manufacturing sector

A taxonomy and state of the art revision on affective games

Affective Games are a sub-field of Affective Computing that tries to study how to design videogames that are able to react to the emotions expressed by the player, as well as provoking desired emotions to them. To achieve those goals it is necessary to research on how to measure and detect human emotions using a computer, and how to adapt videogames to the perceived emotions to finally provoke them to the players. This work presents a taxonomy for research on affective games centring on the aforementioned issues. Here we devise as well a revision of the most relevant published works known to the authors on this area. Finally, we analyse and discuss which important research problem are yet open and might be tackled by future investigations in the area of Affective GamesThis work has been co-funded by the following research projects: EphemeCH (TIN2014-56494-C4-{1,4}-P) and DeepBio (TIN2017-85727-C4-3-P) by Spanish Ministry of Economy and Competitivity, under the European Regional Development Fund FEDER, and Justice Programme of the European Union (2014–2020) 723180 – RiskTrack – JUST-2015-JCOO-AG/JUST-2015-JCOO-AG-

Enhancing video game performance through an individualized biocybernetic system

Biocybernetic systems are physiological software systems that explicitly utilize physiological signals to control or adapt software functionality (Pope et al., 1995.) These systems have tremendous potential for innovation in human computer interaction by using physiological signals to infer a user\u27s emotional and mental states (Allanson & Fairclough, 2004; Fairclough, 2008). Nevertheless, development of these systems has been ultimately hindered by two fundamental challenges. First, these systems make generalizations about physiological indicators of cognitive states across populations when, in fact, relationships between physiological responses and cognitive states are specific to each individual (Andreassi, 2006). Second, they often employ largely inconsistent retrospective techniques to subjectively infer user\u27s mental state (Fairclough, 2008). An individualized biocybernetic system was developed to address the fundamental challenges of biocybernetic research. This system was used to adapt video game difficulty through real-time classifications of physiological responses to subjective appraisals. A study was conducted to determine the system\u27s ability to improve player\u27s performance. The results provide evidence of significant task performance increase and higher attained task difficulty when players interacted with the game using the system than without. This work offers researchers with an alternative method for software adaptation by conforming to the individual characteristics of each user

Psychophysiology-based QoE assessment : a survey

We present a survey of psychophysiology-based assessment for quality of experience (QoE) in advanced multimedia technologies. We provide a classification of methods relevant to QoE and describe related psychological processes, experimental design considerations, and signal analysis techniques. We summarize multimodal techniques and discuss several important aspects of psychophysiology-based QoE assessment, including the synergies with psychophysical assessment and the need for standardized experimental design. This survey is not considered to be exhaustive but serves as a guideline for those interested to further explore this emerging field of research

Cognitive conflict in human–automation interactions: A psychophysiological study

The review of literature in sociology and distributed artificial intelligence reveals that the occurrence of conflict is a remarkable precursor to the disruption of multi-agent systems. The study of this concept could be applied to human factors concerns, as man-system conflict appears to provoke perseveration behavior and to degrade attentional abilities with a trend to excessive focus. Once entangled in such conflicts, the human operator will do anything to succeed in his current goal even if it jeopardizes the mission. In order to confirm these findings, an experimental setup, composed of a real unmanned ground vehicle, a ground station is developed. A scenario involving an authority conflict between the partici- pants and the robot is proposed. Analysis of the effects of the conflict on the participants’ cognition and arousal is assessed through heart-rate measurement (reflecting stress level) and eye-tracking techniques (index of attentional focus). Our results clearly show that the occurrence of the conflict leads to perseveration behavior and can induce higher heart rate as well as excessive attentional focus. These results are discussed in terms of task commitment issues and increased arousal. Moreover, our results suggest that individual differences may predict susceptibility to perseveration behavior