Towards emotional interaction: using movies to automatically learn users’ emotional states

The HCI community is actively seeking novel methodologies to gain insight into the user's experience during interaction with both the application and the content. We propose an emotional recognition engine capable of automatically recognizing a set of human emotional states using psychophysiological measures of the autonomous nervous system, including galvanic skin response, respiration, and heart rate. A novel pattern recognition system, based on discriminant analysis and support vector machine classifiers is trained using movies' scenes selected to induce emotions ranging from the positive to the negative valence dimension, including happiness, anger, disgust, sadness, and fear. In this paper we introduce an emotion recognition system and evaluate its accuracy by presenting the results of an experiment conducted with three physiologic sensors.info:eu-repo/semantics/publishedVersio

Incorporating Cognitive Neuroscience Techniques to Enhance User Experience Research Practices

User Experience (UX) involves every interaction that customers have with products, and it plays a crucial role in determining the success of a product in the market. While there are numerous methods available in literature for assessing UX, they often overlook the emotional aspect of the user\u27s experience. As a result, cognitive neuroscience methods are gaining popularity, but they have certain limitations such as difficulty in collecting neurophysiological data, potential for errors, and lengthy procedures. This article aims to examine the most effective research practices using cognitive neuroscience techniques and develop a standardized procedure for conducting UX research. To achieve this objective, the study conducts a comprehensive review of UX research that employs cognitive neuroscience methods published between 2017 and 2022

Face Emotion Recognition Based on Machine Learning: A Review

Computers can now detect, understand, and evaluate emotions thanks to recent developments in machine learning and information fusion. Researchers across various sectors are increasingly intrigued by emotion identification, utilizing facial expressions, words, body language, and posture as means of discerning an individual's emotions. Nevertheless, the effectiveness of the first three methods may be limited, as individuals can consciously or unconsciously suppress their true feelings. This article explores various feature extraction techniques, encompassing the development of machine learning classifiers like k-nearest neighbour, naive Bayesian, support vector machine, and random forest, in accordance with the established standard for emotion recognition. The paper has three primary objectives: firstly, to offer a comprehensive overview of effective computing by outlining essential theoretical concepts; secondly, to describe in detail the state-of-the-art in emotion recognition at the moment; and thirdly, to highlight important findings and conclusions from the literature, with an emphasis on important obstacles and possible future paths, especially in the creation of state-of-the-art machine learning algorithms for the identification of emotions

Thinking Fast or slow? Understanding Answering Behavior Using Dual-Process Theory through Mouse Cursor Movements

Users’ underlying cognitive states govern their behaviors online. For instance, an extreme cognitive burden during live system use would negatively influence important user behaviors such as using the system and purchasing a product. Thus, inferring the user's cognitive state has practical significance for the commercialized systems. We use Dual-Process Theory to explain how the mouse cursor movements can be an effective measure of cognitive load. In an experimental study with five hundred and thirty-four subjects, we induced cognitive burden then monitored mouse cursor movements when the participants answered questions in an online survey. We found that participants' mouse cursor movements slow down when they are engaged in cognitively demanding tasks. With the newly derived measures, we were able to infer the state of heightened cognitive load with an overall accuracy of 70.22%. The results enable researchers to measure users' cognitive load with more granularity and present a new, theoretically sound method to assess the user's cognitive state

The Usefulness of Multi-Sensor Affect Detection on User Experience: An Application of Biometric Measurement Systems on Online Purchasing

abstract: Traditional usability methods in Human-Computer Interaction (HCI) have been extensively used to understand the usability of products. Measurements of user experience (UX) in traditional HCI studies mostly rely on task performance and observable user interactions with the product or services, such as usability tests, contextual inquiry, and subjective self-report data, including questionnaires, interviews, and usability tests. However, these studies fail to directly reflect a user’s psychological involvement and further fail to explain the cognitive processing and the related emotional arousal. Thus, capturing how users think and feel when they are using a product remains a vital challenge of user experience evaluation studies. Conversely, recent research has revealed that sensor-based affect detection technologies, such as eye tracking, electroencephalography (EEG), galvanic skin response (GSR), and facial expression analysis, effectively capture affective states and physiological responses. These methods are efficient indicators of cognitive involvement and emotional arousal and constitute effective strategies for a comprehensive measurement of UX. The literature review shows that the impacts of sensor-based affect detection systems to the UX can be categorized in two groups: (1) confirmatory to validate the results obtained from the traditional usability methods in UX evaluations; and (2) complementary to enhance the findings or provide more precise and valid evidence. Both provided comprehensive findings to uncover the issues related to mental and physiological pathways to enhance the design of product and services. Therefore, this dissertation claims that it can be efficient to integrate sensor-based affect detection technologies to solve the current gaps or weaknesses of traditional usability methods. The dissertation revealed that the multi-sensor-based UX evaluation approach through biometrics tools and software corroborated user experience identified by traditional UX methods during an online purchasing task. The use these systems enhanced the findings and provided more precise and valid evidence to predict the consumer purchasing preferences. Thus, their impact was “complementary” on overall UX evaluation. The dissertation also provided information of the unique contributions of each tool and recommended some ways user experience researchers can combine both sensor-based and traditional UX approaches to explain consumer purchasing preferences.Dissertation/ThesisDoctoral Dissertation Human Systems Engineering 201