Automatic detection of affect and cognitive load from multimodal information

The affective and cognitive states of a computer user have significant impact on how the interaction is experienced. Therefore, understanding these aspects of human psychology is of great importance to improve the design of intelligent systems. This thesis investigates the automatic detection of affect and cognitive load (CL) from multimodal information. Affect detection is evaluated from data collected during controlled stimulus presentation and naturalistic interactions with an Intelligent Tutoring System (ITS). This thesis contributes to evidence that pattern recognition techniques, applied to a single, and even more so, to a combination of modalities, can lead to accurate automatic affect and cognitive load detection. There is considerable motivation for measuring affect from physiological signals, particularly because it is robust against social masking and suitable for reflecting inner feelings. This thesis evaluates techniques to improve affect detection accuracies from physiology. Analysis shows that affect detection is more accurate in controlled interactions with slightly lowers accuracies in naturalistic interactions. It is useful to consider other modalities, for example, behavioral channels to improve accuracies and for practicality. Therefore, facial features are considered together with physiology. Combined classifiers are evaluated, proving to be more accurate than single classifiers at detecting affects from multimodal features. Both categorical and dimensional representations of affect are evaluated during ITS interactions. Affect and CL can be considered as two aspects of a single experience; therefore, this thesis proposes a single computational framework for evaluating both. Results indicate that face is more suitable for detecting affect, but physiology proved to be more suitable for CL. However, the highest accuracies were achieved from multimodal fusion. Subject-independent models proved to be more suitable for CL detection