An Evaluation of Mouse and Keyboard Interaction Indicators towards Non-intrusive and Low Cost Affective Modeling in an Educational Context

AbstractIn this paper we propose a series of indicators, which derive from user's interactions with mouse and keyboard. The goal is to evaluate their use in identifying affective states and behavior changes in an e-learning platform by means of non-intrusive and low cost methods. The approach we have followed study user's interactions regardless of the task being performed and its presentation, aiming at finding a solution applicable in any domain. In particular, mouse movements and clicks, as well as keystrokes were recorded during a math problem solving activity where users involved in the experiment had not only to score their degree of valence (i.e., pleasure versus displeasure) and arousal (i.e., high activation versus low activation) of their affective states after each problem by using the Self-Assessment-Manikin scale, but also type a description of their own feelings. By using that affective labeling, we evaluated the information provided by these different indicators processed from the original user's interactions logs. In total, we computed 42 keyboard indicators and 96 mouse indicators

Exploring the dynamic interplay of cognitive load and emotional arousal by using multimodal measurements: Correlation of pupil diameter and emotional arousal in emotionally engaging tasks

Multimodal data analysis and validation based on streams from state-of-the-art sensor technology such as eye-tracking or emotion recognition using the Facial Action Coding System (FACTs) with deep learning allows educational researchers to study multifaceted learning and problem-solving processes and to improve educational experiences. This study aims to investigate the correlation between two continuous sensor streams, pupil diameter as an indicator of cognitive workload and FACTs with deep learning as an indicator of emotional arousal (RQ 1a), specifically for epochs of high, medium, and low arousal (RQ 1b). Furthermore, the time lag between emotional arousal and pupil diameter data will be analyzed (RQ 2). 28 participants worked on three cognitively demanding and emotionally engaging everyday moral dilemmas while eye-tracking and emotion recognition data were collected. The data were pre-processed in Phyton (synchronization, blink control, downsampling) and analyzed using correlation analysis and Granger causality tests. The results show negative and statistically significant correlations between the data streams for emotional arousal and pupil diameter. However, the correlation is negative and significant only for epochs of high arousal, while positive but non-significant relationships were found for epochs of medium or low arousal. The average time lag for the relationship between arousal and pupil diameter was 2.8 ms. In contrast to previous findings without a multimodal approach suggesting a positive correlation between the constructs, the results contribute to the state of research by highlighting the importance of multimodal data validation and research on convergent vagility. Future research should consider emotional regulation strategies and emotional valence.Comment: The first two authors contributed equally to the manuscrip

The Impact of Emotional Sounds on Arousal and Task Performance

In times of emotional arousal, it is hypothesized that neural processes are triggered to “heighten” our senses to better respond to threatening stimuli. Some studies have tested this by exposing participants to emotional sounds to determine their impacts on visual acuity but have found mixed results. Previous studies have not investigated interactions between arousal induced by emotional sounds and visual acuity. Participants (N = 42) performed an orientation detection task while presented in silence or with sounds that varied in valence. Results displayed comparable accuracy across conditions but significantly faster response times during the presentation of negative sounds on the opposite side of the Gabor patch compared to neutral sounds irrespective of spatial location. Additionally, pupil size was significantly greater in the negative condition than in the neutral condition. These findings delineate how changes in arousal due to environmental factors can lead to changes in human performance

Neuromarketing: an eye-tracking experimental study

openNel presente studio si è cercato di capire, utilizzando i movimenti oculari, quali sono le differenze nel modo di osservare e di valutare le confezioni di un determinato prodotto, in questo caso di caffè, in relazione alla sua complessità, ovvero alla quantità di elementi al suo interno, ed in relazione al brand, che può risultare noto o meno all’osservatore. Gli aspetti presi in considerazione sono i tempi di reazione nella risposta in merito alla conoscenza del brand, i prezzi attribuiti a ciascuna confezione, la dilatazione pupillare, i tempi di fissazione, e la modalità di esplorazione di ciascuna confezione.In this study we tried to understand, using eye movements, what are the differences in the way of observing and evaluating the packaging of a specific product, in this case coffee, in relation to its complexity, i.e. the quantity of elements within it, and in relation to the brand, which may or may not be known by the observer. The aspects taken into consideration are the reaction times in the response regarding brand awareness, the prices attributed to each pack, pupil dilation, fixation times, and the way each pack is explored

Survey of contributions for a pipeline of emotion recognition and awareness - context variables, instruments & sensors, pre-processing techniques and extracted properties for automatic recognition of emotions

A avaliação emocional tem sido uma área de investigação, desde há muitos anos, na área da saúde e na área psicossocial. Foi a partir da década de 90 que o reconhecimento de emoções ganhou mais atenção por parte dos investigadores, tornando-se num importante tópico de investigação até aos dias de hoje (Basu, Bag, Mahadevappa, Mukherjee, & Guha, 2016). Segundo Picard, o estudo das emoções moveu-se da psicologia para a área da computação, criando um novo campo de investigação chamado de Affective Computing (AC). Aliás, no seu livro “Affective Computing”, indica as bases para a criação de um sistema inteligente para deteção emocional de forma automática (R. W. Picard, 1995). Nos últimos anos, tem-se presenciado a um aumento deste tipo de investigações, talvez pela necessidade de transformar a relação entre as coisas (e.g. hardware, software e produtos em geral) e as pessoas, numa interação mais inteligente e natural (R. Picard & Klein, 2002), transformando assim o AC num tópico importante de investigação (Bos, 2010). Vários autores consideram que a deteção automática de emoções poderá ter um impacto positivo na vida das pessoas. Por exemplo, a área da psicologia poderá beneficiar, com menos subjetividade, de dados contínuos e menos diferidos no tempo; a saúde poderá ser avaliada com informação complementar à fisiológica; poderá ser mais fácil detetar delitos como atos de delinquência e atentados terroristas; e será mais fácil desenhar produtos especializados em provocar ou transmitir emoções no mundo virtual (Murad & Malkawi, 2012). Poderá também ser possível criar sistemas inteligentes do ponto de vista afetivo, conscientes ao nível emocional, capazes de percecionar e reagir às emoções dos utilizadores. Apesar de existirem já vários estudos com o objetivo de detetar automaticamente emoções, os autores acreditam que a correlação de variáveis sociais, culturais e religiosas, com as fisiológicas, poderá contribuir de forma positiva para a qualidade dos resultados obtidos. Neste contexto, está-se a preparar uma experiência para detetar automaticamente o bem-estar nos trabalhadores de escritório. Pretende-se recolher variáveis de contexto de várias modalidades e, depois do respetivo pré-processamento, usar esses dados como input de algoritmos de Machine Learning (ML) para a respetiva classificação. O objetivo é verificar a possibilidade de criar sistemas inteligentes do ponto de vista afetivo, conscientes ao nível emocional, capazes de percecionar e reagir às emoções dos funcionários de escritório. Este relatório resume as obras estudadas pelos autores na área do AC na revisão bibliográfica sobre o tema. Sugere-se um sistema de tokens para melhor categorização da informação, e propõe-se também uma sistematização da informação através da organização desses tokens em quadros resumo, para permitir uma análise agregada das investigações. Na secção seguinte são resumidas as variáveis de contexto e propriedades de domínio utilizadas pelos autores. Depois são apresentados os instrumentos & sensores utilizados na recolha das variáveis de contexto. Posteriormente são resumidas as técnicas de pré-processamento utilizadas. Conclui-se com uma enumeração das propriedades extraídas mais utilizadas nas obras estudadas.Emotional assessment has been a research area of health and psychosocial field, since many years. It was from 90’s that the recognition of emotions gained more attention from the researchers, becoming an importante topic of research up to today (Basu, Bag, Mahadevappa, Mukherjee, & Guha, 2016). According to Picard, the study of emotions moved from psychology to the area of computing, creating a new research field called Affective Computing (AC). In fact, in her book “Affective Computing”, she indicates the basis for creating na intelligent system for automatic emotional detection (R. W. Picard, 1995). In recent years, there has been an increase in this kind of research, perhaps due the need to transform the interaction between things (e.g. hardware, software and products in general) and people more natural and intelligent (R. Picard & Klein, 2002). This transformed the AC in an important research topic (Bos, 2010). Several authors believe that the automatic emotional detection can have positive impacto on people’s lives. As an exemple, the area of psychology may benefit with less subjectivity, continuous and less deferred data in time; health can be assessed with additional info besides physiological data; it may be easier to detect crimes such as acts of delinquency and terrorist attacks; and it will be easier to design products specialized in provoking or transmitting emotions in the virtual world (Murad & Malkawi, 2012). It may also be possible to create intelligent affective systems. Emotion-aware systems that can understanding and react to people emotions. Although there are already several studies with the objective of automatically detecting emotions, the authors believe that the correlation of social, cultural and religious variables with physiological ones, may contribute positively to the quality of the results obtained. In this context, an experiment is being prepared to automatically detect the well-being of office workers. It is intended to collect context variables of several modalities and, after the pre- processing phase, use that data as input to Machine Learning (ML) classification algorithms. The goal is to verify the possibility of creating intelligent systems from an affective point of view, conscious at the emotional level, capable of perceiving and reacting to the emotions of office workers. This technical report summarizes the studied researchs by the authors during the bibliographic review on the AC topic. A token system is suggested for better categorization of information, and a systematization of information is also proposed through the organization of these tokens in summary tables, to allow an aggregated analysis of the investigations. The following section summarizes the context variables and domain properties used by the authors. Then, the instruments & sensors used to collect the context variables are presented. Subsequently, the pre-processing techniques used are summarized. It concludes with an enumeration of the extracted properties most used in the studied works.info:eu-repo/semantics/draf