Multimodality Inferring of Human Cognitive States Based on Integration of Neuro-Fuzzy Network and Information Fusion Techniques

To achieve an effective and safe operation on the machine system where the human interacts with the machine mutually, there is a need for the machine to understand the human state, especially cognitive state, when the human's operation task demands an intensive cognitive activity. Due to a well-known fact with the human being, a highly uncertain cognitive state and behavior as well as expressions or cues, the recent trend to infer the human state is to consider multimodality features of the human operator. In this paper, we present a method for multimodality inferring of human cognitive states by integrating neuro-fuzzy network and information fusion techniques. To demonstrate the effectiveness of this method, we take the driver fatigue detection as an example. The proposed method has, in particular, the following new features. First, human expressions are classified into four categories: (i) casual or contextual feature, (ii) contact feature, (iii) contactless feature, and (iv) performance feature. Second, the fuzzy neural network technique, in particular Takagi-Sugeno-Kang (TSK) model, is employed to cope with uncertain behaviors. Third, the sensor fusion technique, in particular ordered weighted aggregation (OWA), is integrated with the TSK model in such a way that cues are taken as inputs to the TSK model, and then the outputs of the TSK are fused by the OWA which gives outputs corresponding to particular cognitive states under interest (e.g., fatigue). We call this method TSK-OWA. Validation of the TSK-OWA, performed in the Northeastern University vehicle drive simulator, has shown that the proposed method is promising to be a general tool for human cognitive state inferring and a special tool for the driver fatigue detection

Fatigue effect on task performance in haptic virtual environment for home-based rehabilitation

Stroke rehabilitation is to train the motor function of a patient’s limb. In this process, functional assessment is of importance, and it is primarily based on a patient’s task performance. The context of the rehabilitation discussed in this thesis is such that functional assessment is conducted through a computer system and the Internet. In particular, a patient performs the task at home in a haptic virtual environment, and the task performance is transmitted to the therapist over the Internet. One problem with this approach to functional assessment is that a patient’s mind state is little known to the therapist. This immediately leads to one question, that is, whether an elevated mind state will have some significant effect on the patient’s task performance? If so, this approach can result in a considerable error. The overall objective of this thesis study was to generate an answer to the aforementioned question. The study focused on a patient’s elevated fatigue state. The specific objectives of the study include: (i) developing a haptic virtual environment prototype system for functional assessment, (ii) developing a physiological-based inference system for fatigue state, and (iii) performing an experiment to generate knowledge regarding the fatigue effect on task performance. With a limited resource in recruiting patients in the experiment, the study conducted few experiments on patients but mostly on healthy subjects. The study has concluded: (1) the proposed haptic virtual environment system is effective for the wrist coordination task and is likely promising to other tasks, (2) the accuracy of proposed fatigue inference system achieves 89.54%, for two levels of fatigue state, which is promising, (3) the elevated fatigue state significantly affects task performance in the context of wrist coordination task, and (4) the accuracy of the individual-based inference approach is significantly higher than that of the group-based inference approach. The main contributions of the thesis are (1) generation of the new knowledge regarding the fatigue effect on task performance in the context of home-based rehabilitation, (2) provision of the new fatigue inference system with the highest accuracy in comparison with the existing approaches in literature, and (3) generation of the new knowledge regarding the difference between the individual-based inference and group-based inference approaches